Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250815 01:52:34.453498 427 test_util.cc:276] Using random seed: -1530402989
W20250815 01:52:35.683128 427 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.185s user 0.433s sys 0.751s
W20250815 01:52:35.683492 427 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.185s user 0.433s sys 0.751s
I20250815 01:52:35.685840 427 ts_itest-base.cc:115] Starting cluster with:
I20250815 01:52:35.686080 427 ts_itest-base.cc:116] --------------
I20250815 01:52:35.686283 427 ts_itest-base.cc:117] 4 tablet servers
I20250815 01:52:35.686532 427 ts_itest-base.cc:118] 3 replicas per TS
I20250815 01:52:35.686730 427 ts_itest-base.cc:119] --------------
2025-08-15T01:52:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T01:52:35Z Disabled control of system clock
I20250815 01:52:35.729993 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:39275
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:39275 with env {}
W20250815 01:52:36.034835 441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:36.035434 441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:52:36.035895 441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:36.066465 441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:52:36.066782 441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:52:36.067039 441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:52:36.067287 441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:52:36.102294 441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:39275
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:52:36.103610 441 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:52:36.105199 441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:52:36.116518 447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:36.117683 448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:37.316972 449 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1198 milliseconds
W20250815 01:52:37.317034 441 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.201s user 0.394s sys 0.801s
W20250815 01:52:37.317577 441 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.201s user 0.395s sys 0.802s
W20250815 01:52:37.318010 450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:52:37.318079 441 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:52:37.319768 441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:52:37.323050 441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:52:37.324584 441 hybrid_clock.cc:648] HybridClock initialized: now 1755222757324505 us; error 68 us; skew 500 ppm
I20250815 01:52:37.325747 441 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:52:37.335242 441 webserver.cc:480] Webserver started at http://127.0.106.254:40745/ using document root <none> and password file <none>
I20250815 01:52:37.336624 441 fs_manager.cc:362] Metadata directory not provided
I20250815 01:52:37.336932 441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:52:37.337579 441 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:52:37.344810 441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5603f3984aec4573b3407256b696508f"
format_stamp: "Formatted at 2025-08-15 01:52:37 on dist-test-slave-h491"
I20250815 01:52:37.346422 441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5603f3984aec4573b3407256b696508f"
format_stamp: "Formatted at 2025-08-15 01:52:37 on dist-test-slave-h491"
I20250815 01:52:37.356079 441 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20250815 01:52:37.364444 457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:37.365629 441 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 01:52:37.366027 441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5603f3984aec4573b3407256b696508f"
format_stamp: "Formatted at 2025-08-15 01:52:37 on dist-test-slave-h491"
I20250815 01:52:37.366545 441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:52:37.455768 441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:52:37.457834 441 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:52:37.458565 441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:52:37.539604 441 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:39275
I20250815 01:52:37.539701 508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:39275 every 8 connection(s)
I20250815 01:52:37.542394 441 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:52:37.547977 509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:52:37.552507 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 441
I20250815 01:52:37.553061 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 01:52:37.570191 509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Bootstrap starting.
I20250815 01:52:37.576289 509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Neither blocks nor log segments found. Creating new log.
I20250815 01:52:37.578123 509 log.cc:826] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Log is configured to *not* fsync() on all Append() calls
I20250815 01:52:37.583905 509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: No bootstrap required, opened a new log
I20250815 01:52:37.605727 509 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } }
I20250815 01:52:37.606331 509 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:52:37.606575 509 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5603f3984aec4573b3407256b696508f, State: Initialized, Role: FOLLOWER
I20250815 01:52:37.607281 509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } }
I20250815 01:52:37.607761 509 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:52:37.607987 509 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:52:37.608245 509 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:52:37.612567 509 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } }
I20250815 01:52:37.613308 509 leader_election.cc:304] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5603f3984aec4573b3407256b696508f; no voters:
I20250815 01:52:37.615396 509 leader_election.cc:290] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:52:37.616071 514 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:52:37.618727 514 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 1 LEADER]: Becoming Leader. State: Replica: 5603f3984aec4573b3407256b696508f, State: Running, Role: LEADER
I20250815 01:52:37.619454 514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } }
I20250815 01:52:37.620414 509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:52:37.628832 516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5603f3984aec4573b3407256b696508f. Latest consensus state: current_term: 1 leader_uuid: "5603f3984aec4573b3407256b696508f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } } }
I20250815 01:52:37.629473 516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: This master's current role is: LEADER
I20250815 01:52:37.629467 515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5603f3984aec4573b3407256b696508f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } } }
I20250815 01:52:37.631078 515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: This master's current role is: LEADER
I20250815 01:52:37.634694 522 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:52:37.645406 522 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:52:37.663290 522 catalog_manager.cc:1349] Generated new cluster ID: d8ecc26b485f42e4b33fa89c6233f6d6
I20250815 01:52:37.663550 522 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:52:37.679332 522 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:52:37.680850 522 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:52:37.696210 522 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Generated new TSK 0
I20250815 01:52:37.697340 522 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 01:52:37.713824 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:39275
--builtin_ntp_servers=127.0.106.212:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 01:52:38.029539 533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:38.030056 533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:52:38.030622 533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:38.062568 533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:52:38.063467 533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:52:38.099726 533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:39275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:52:38.101032 533 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:52:38.102564 533 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:52:38.115931 539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:39.519235 538 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 533
W20250815 01:52:39.893965 533 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.777s user 0.642s sys 1.037s
W20250815 01:52:39.895741 533 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.779s user 0.642s sys 1.037s
W20250815 01:52:38.116986 540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:39.896368 541 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1777 milliseconds
W20250815 01:52:39.897461 542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:52:39.897459 533 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:52:39.902089 533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:52:39.904757 533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:52:39.906275 533 hybrid_clock.cc:648] HybridClock initialized: now 1755222759906225 us; error 49 us; skew 500 ppm
I20250815 01:52:39.907367 533 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:52:39.915316 533 webserver.cc:480] Webserver started at http://127.0.106.193:34735/ using document root <none> and password file <none>
I20250815 01:52:39.916597 533 fs_manager.cc:362] Metadata directory not provided
I20250815 01:52:39.916882 533 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:52:39.917457 533 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:52:39.924113 533 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "100a2f6a32f7408886e9b6a5cabdc213"
format_stamp: "Formatted at 2025-08-15 01:52:39 on dist-test-slave-h491"
I20250815 01:52:39.925652 533 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "100a2f6a32f7408886e9b6a5cabdc213"
format_stamp: "Formatted at 2025-08-15 01:52:39 on dist-test-slave-h491"
I20250815 01:52:39.935191 533 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.002s sys 0.006s
I20250815 01:52:39.943291 549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:39.944501 533 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20250815 01:52:39.944882 533 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "100a2f6a32f7408886e9b6a5cabdc213"
format_stamp: "Formatted at 2025-08-15 01:52:39 on dist-test-slave-h491"
I20250815 01:52:39.945334 533 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:52:40.013832 533 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:52:40.015784 533 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:52:40.016357 533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:52:40.020031 533 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:52:40.026060 533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:52:40.026335 533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:40.026657 533 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:52:40.026866 533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:40.229408 533 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:34581
I20250815 01:52:40.229497 661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:34581 every 8 connection(s)
I20250815 01:52:40.232039 533 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:52:40.235080 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 533
I20250815 01:52:40.235554 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 01:52:40.248868 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:39275
--builtin_ntp_servers=127.0.106.212:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:52:40.273550 662 heartbeater.cc:344] Connected to a master server at 127.0.106.254:39275
I20250815 01:52:40.274001 662 heartbeater.cc:461] Registering TS with master...
I20250815 01:52:40.275070 662 heartbeater.cc:507] Master 127.0.106.254:39275 requested a full tablet report, sending...
I20250815 01:52:40.277460 474 ts_manager.cc:194] Registered new tserver with Master: 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581)
I20250815 01:52:40.279976 474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:53775
W20250815 01:52:40.590329 666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:40.590853 666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:52:40.591307 666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:40.622099 666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:52:40.623152 666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:52:40.657428 666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:39275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:52:40.658677 666 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:52:40.660213 666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:52:40.672649 672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:52:41.283291 662 heartbeater.cc:499] Master 127.0.106.254:39275 was elected leader, sending a full tablet report...
W20250815 01:52:42.076591 671 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 666
W20250815 01:52:42.449191 666 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.776s user 0.629s sys 1.146s
W20250815 01:52:42.449600 666 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.777s user 0.629s sys 1.146s
W20250815 01:52:40.672859 673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:42.451570 675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:42.454107 674 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1775 milliseconds
I20250815 01:52:42.454134 666 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:52:42.455312 666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:52:42.457792 666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:52:42.459153 666 hybrid_clock.cc:648] HybridClock initialized: now 1755222762459112 us; error 45 us; skew 500 ppm
I20250815 01:52:42.459939 666 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:52:42.466223 666 webserver.cc:480] Webserver started at http://127.0.106.194:39101/ using document root <none> and password file <none>
I20250815 01:52:42.467172 666 fs_manager.cc:362] Metadata directory not provided
I20250815 01:52:42.467384 666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:52:42.467831 666 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:52:42.472131 666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f046314d04e3419db0bc3c06aae3b292"
format_stamp: "Formatted at 2025-08-15 01:52:42 on dist-test-slave-h491"
I20250815 01:52:42.473275 666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f046314d04e3419db0bc3c06aae3b292"
format_stamp: "Formatted at 2025-08-15 01:52:42 on dist-test-slave-h491"
I20250815 01:52:42.480290 666 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250815 01:52:42.486467 682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:42.487706 666 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250815 01:52:42.488047 666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f046314d04e3419db0bc3c06aae3b292"
format_stamp: "Formatted at 2025-08-15 01:52:42 on dist-test-slave-h491"
I20250815 01:52:42.488394 666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:52:42.539173 666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:52:42.540596 666 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:52:42.541014 666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:52:42.544072 666 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:52:42.548631 666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:52:42.548821 666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:42.549082 666 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:52:42.549253 666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:42.700287 666 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:42751
I20250815 01:52:42.700398 794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:42751 every 8 connection(s)
I20250815 01:52:42.702937 666 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 01:52:42.707882 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 666
I20250815 01:52:42.708390 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 01:52:42.714557 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:39275
--builtin_ntp_servers=127.0.106.212:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:52:42.725499 795 heartbeater.cc:344] Connected to a master server at 127.0.106.254:39275
I20250815 01:52:42.726082 795 heartbeater.cc:461] Registering TS with master...
I20250815 01:52:42.728073 795 heartbeater.cc:507] Master 127.0.106.254:39275 requested a full tablet report, sending...
I20250815 01:52:42.730479 474 ts_manager.cc:194] Registered new tserver with Master: f046314d04e3419db0bc3c06aae3b292 (127.0.106.194:42751)
I20250815 01:52:42.731717 474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:49119
W20250815 01:52:43.022333 799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:43.022933 799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:52:43.023439 799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:43.054047 799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:52:43.054955 799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:52:43.089057 799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:39275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:52:43.090400 799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:52:43.091924 799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:52:43.105085 805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:52:43.735611 795 heartbeater.cc:499] Master 127.0.106.254:39275 was elected leader, sending a full tablet report...
W20250815 01:52:43.111359 808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:43.106292 806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:44.300897 807 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 01:52:44.300943 799 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:52:44.305356 799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:52:44.308039 799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:52:44.309387 799 hybrid_clock.cc:648] HybridClock initialized: now 1755222764309346 us; error 47 us; skew 500 ppm
I20250815 01:52:44.310161 799 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:52:44.316445 799 webserver.cc:480] Webserver started at http://127.0.106.195:41001/ using document root <none> and password file <none>
I20250815 01:52:44.317343 799 fs_manager.cc:362] Metadata directory not provided
I20250815 01:52:44.317561 799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:52:44.318022 799 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:52:44.322558 799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
format_stamp: "Formatted at 2025-08-15 01:52:44 on dist-test-slave-h491"
I20250815 01:52:44.323657 799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
format_stamp: "Formatted at 2025-08-15 01:52:44 on dist-test-slave-h491"
I20250815 01:52:44.330529 799 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250815 01:52:44.336386 815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:44.337464 799 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250815 01:52:44.337810 799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
format_stamp: "Formatted at 2025-08-15 01:52:44 on dist-test-slave-h491"
I20250815 01:52:44.338131 799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:52:44.395685 799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:52:44.397158 799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:52:44.397604 799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:52:44.400620 799 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:52:44.404578 799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:52:44.404786 799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:44.405026 799 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:52:44.405184 799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:44.563215 799 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:42383
I20250815 01:52:44.563309 928 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:42383 every 8 connection(s)
I20250815 01:52:44.565788 799 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 01:52:44.568907 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 799
I20250815 01:52:44.569530 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 01:52:44.579106 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.196:0
--local_ip_for_outbound_sockets=127.0.106.196
--webserver_interface=127.0.106.196
--webserver_port=0
--tserver_master_addrs=127.0.106.254:39275
--builtin_ntp_servers=127.0.106.212:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:52:44.588173 929 heartbeater.cc:344] Connected to a master server at 127.0.106.254:39275
I20250815 01:52:44.588617 929 heartbeater.cc:461] Registering TS with master...
I20250815 01:52:44.589637 929 heartbeater.cc:507] Master 127.0.106.254:39275 requested a full tablet report, sending...
I20250815 01:52:44.591915 474 ts_manager.cc:194] Registered new tserver with Master: dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195:42383)
I20250815 01:52:44.593241 474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:49543
W20250815 01:52:44.887818 933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:44.888445 933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:52:44.888998 933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:44.920418 933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:52:44.921276 933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.196
I20250815 01:52:44.960966 933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.106.196
--webserver_port=0
--tserver_master_addrs=127.0.106.254:39275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.196
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:52:44.962248 933 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:52:44.963874 933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:52:44.977516 939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:52:45.596740 929 heartbeater.cc:499] Master 127.0.106.254:39275 was elected leader, sending a full tablet report...
W20250815 01:52:44.978615 940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:46.181617 942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:46.184029 941 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1201 milliseconds
I20250815 01:52:46.184158 933 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:52:46.185317 933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:52:46.188022 933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:52:46.189441 933 hybrid_clock.cc:648] HybridClock initialized: now 1755222766189401 us; error 56 us; skew 500 ppm
I20250815 01:52:46.190207 933 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:52:46.196810 933 webserver.cc:480] Webserver started at http://127.0.106.196:42817/ using document root <none> and password file <none>
I20250815 01:52:46.197669 933 fs_manager.cc:362] Metadata directory not provided
I20250815 01:52:46.197866 933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:52:46.198294 933 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:52:46.202823 933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4884b647203a42c18c5875acba819b78"
format_stamp: "Formatted at 2025-08-15 01:52:46 on dist-test-slave-h491"
I20250815 01:52:46.203919 933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4884b647203a42c18c5875acba819b78"
format_stamp: "Formatted at 2025-08-15 01:52:46 on dist-test-slave-h491"
I20250815 01:52:46.210932 933 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250815 01:52:46.216517 950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:46.217587 933 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.002s
I20250815 01:52:46.217895 933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4884b647203a42c18c5875acba819b78"
format_stamp: "Formatted at 2025-08-15 01:52:46 on dist-test-slave-h491"
I20250815 01:52:46.218192 933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:52:46.271164 933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:52:46.272619 933 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:52:46.273013 933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:52:46.275475 933 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:52:46.279544 933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:52:46.279758 933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:46.279963 933 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:52:46.280100 933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:46.430050 933 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.196:35561
I20250815 01:52:46.430153 1062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.196:35561 every 8 connection(s)
I20250815 01:52:46.432672 933 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250815 01:52:46.442909 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 933
I20250815 01:52:46.443327 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250815 01:52:46.459321 1063 heartbeater.cc:344] Connected to a master server at 127.0.106.254:39275
I20250815 01:52:46.459802 1063 heartbeater.cc:461] Registering TS with master...
I20250815 01:52:46.460978 1063 heartbeater.cc:507] Master 127.0.106.254:39275 requested a full tablet report, sending...
I20250815 01:52:46.463532 473 ts_manager.cc:194] Registered new tserver with Master: 4884b647203a42c18c5875acba819b78 (127.0.106.196:35561)
I20250815 01:52:46.465157 473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.196:38599
I20250815 01:52:46.478035 427 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250815 01:52:46.521479 473 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36176:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 01:52:46.614846 864 tablet_service.cc:1468] Processing CreateTablet for tablet 7d08a1abafe2478ab98e528d66854315 (DEFAULT_TABLE table=TestTable [id=33cc433e1f8241e290bf1021de91287a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:52:46.616801 864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d08a1abafe2478ab98e528d66854315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:52:46.616434 730 tablet_service.cc:1468] Processing CreateTablet for tablet 7d08a1abafe2478ab98e528d66854315 (DEFAULT_TABLE table=TestTable [id=33cc433e1f8241e290bf1021de91287a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:52:46.618217 730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d08a1abafe2478ab98e528d66854315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:52:46.622016 597 tablet_service.cc:1468] Processing CreateTablet for tablet 7d08a1abafe2478ab98e528d66854315 (DEFAULT_TABLE table=TestTable [id=33cc433e1f8241e290bf1021de91287a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:52:46.623839 597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d08a1abafe2478ab98e528d66854315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:52:46.643849 1082 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4: Bootstrap starting.
I20250815 01:52:46.646675 1083 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: Bootstrap starting.
I20250815 01:52:46.653607 1082 tablet_bootstrap.cc:654] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4: Neither blocks nor log segments found. Creating new log.
I20250815 01:52:46.654404 1083 tablet_bootstrap.cc:654] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: Neither blocks nor log segments found. Creating new log.
I20250815 01:52:46.656327 1084 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213: Bootstrap starting.
I20250815 01:52:46.656993 1082 log.cc:826] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4: Log is configured to *not* fsync() on all Append() calls
I20250815 01:52:46.657379 1083 log.cc:826] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: Log is configured to *not* fsync() on all Append() calls
I20250815 01:52:46.662717 1083 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: No bootstrap required, opened a new log
I20250815 01:52:46.663151 1083 ts_tablet_manager.cc:1397] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20250815 01:52:46.663252 1082 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4: No bootstrap required, opened a new log
I20250815 01:52:46.664290 1082 ts_tablet_manager.cc:1397] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.010s
I20250815 01:52:46.665045 1084 tablet_bootstrap.cc:654] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213: Neither blocks nor log segments found. Creating new log.
I20250815 01:52:46.667539 1084 log.cc:826] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213: Log is configured to *not* fsync() on all Append() calls
I20250815 01:52:46.675091 1084 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213: No bootstrap required, opened a new log
I20250815 01:52:46.675621 1084 ts_tablet_manager.cc:1397] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.015s
I20250815 01:52:46.684199 1083 raft_consensus.cc:357] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.685200 1083 raft_consensus.cc:383] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:52:46.685519 1083 raft_consensus.cc:738] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f046314d04e3419db0bc3c06aae3b292, State: Initialized, Role: FOLLOWER
I20250815 01:52:46.686527 1083 consensus_queue.cc:260] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.691305 1082 raft_consensus.cc:357] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.692262 1082 raft_consensus.cc:383] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:52:46.692600 1082 raft_consensus.cc:738] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc882f5ec3c742dcb8b2b84270a388f4, State: Initialized, Role: FOLLOWER
I20250815 01:52:46.693331 1083 ts_tablet_manager.cc:1428] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: Time spent starting tablet: real 0.030s user 0.018s sys 0.011s
I20250815 01:52:46.693540 1082 consensus_queue.cc:260] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.698949 1082 ts_tablet_manager.cc:1428] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4: Time spent starting tablet: real 0.034s user 0.026s sys 0.008s
I20250815 01:52:46.702766 1084 raft_consensus.cc:357] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.703821 1084 raft_consensus.cc:383] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:52:46.704197 1084 raft_consensus.cc:738] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100a2f6a32f7408886e9b6a5cabdc213, State: Initialized, Role: FOLLOWER
I20250815 01:52:46.706079 1084 consensus_queue.cc:260] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.711714 1084 ts_tablet_manager.cc:1428] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213: Time spent starting tablet: real 0.036s user 0.030s sys 0.005s
W20250815 01:52:46.712332 796 tablet.cc:2378] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:52:46.730302 1090 raft_consensus.cc:491] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:52:46.730870 1090 raft_consensus.cc:513] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.734278 1090 leader_election.cc:290] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f046314d04e3419db0bc3c06aae3b292 (127.0.106.194:42751), dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195:42383)
I20250815 01:52:46.746387 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "100a2f6a32f7408886e9b6a5cabdc213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f046314d04e3419db0bc3c06aae3b292" is_pre_election: true
I20250815 01:52:46.746702 884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "100a2f6a32f7408886e9b6a5cabdc213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" is_pre_election: true
I20250815 01:52:46.747284 750 raft_consensus.cc:2466] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100a2f6a32f7408886e9b6a5cabdc213 in term 0.
I20250815 01:52:46.747407 884 raft_consensus.cc:2466] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100a2f6a32f7408886e9b6a5cabdc213 in term 0.
I20250815 01:52:46.748550 553 leader_election.cc:304] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100a2f6a32f7408886e9b6a5cabdc213, dc882f5ec3c742dcb8b2b84270a388f4; no voters:
I20250815 01:52:46.749334 1090 raft_consensus.cc:2802] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 01:52:46.749598 1090 raft_consensus.cc:491] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:52:46.749820 1090 raft_consensus.cc:3058] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:52:46.754370 1090 raft_consensus.cc:513] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.756279 1090 leader_election.cc:290] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [CANDIDATE]: Term 1 election: Requested vote from peers f046314d04e3419db0bc3c06aae3b292 (127.0.106.194:42751), dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195:42383)
I20250815 01:52:46.757129 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "100a2f6a32f7408886e9b6a5cabdc213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f046314d04e3419db0bc3c06aae3b292"
I20250815 01:52:46.757303 884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "100a2f6a32f7408886e9b6a5cabdc213" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
I20250815 01:52:46.757556 750 raft_consensus.cc:3058] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:52:46.757762 884 raft_consensus.cc:3058] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 0 FOLLOWER]: Advancing to term 1
W20250815 01:52:46.761725 663 tablet.cc:2378] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:52:46.762651 750 raft_consensus.cc:2466] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100a2f6a32f7408886e9b6a5cabdc213 in term 1.
I20250815 01:52:46.763334 884 raft_consensus.cc:2466] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100a2f6a32f7408886e9b6a5cabdc213 in term 1.
I20250815 01:52:46.764165 550 leader_election.cc:304] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100a2f6a32f7408886e9b6a5cabdc213, f046314d04e3419db0bc3c06aae3b292; no voters:
I20250815 01:52:46.764837 1090 raft_consensus.cc:2802] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:52:46.766870 1090 raft_consensus.cc:695] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [term 1 LEADER]: Becoming Leader. State: Replica: 100a2f6a32f7408886e9b6a5cabdc213, State: Running, Role: LEADER
I20250815 01:52:46.767846 1090 consensus_queue.cc:237] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:46.778812 473 catalog_manager.cc:5582] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 reported cstate change: term changed from 0 to 1, leader changed from <none> to 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193). New cstate: current_term: 1 leader_uuid: "100a2f6a32f7408886e9b6a5cabdc213" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } health_report { overall_health: UNKNOWN } } }
I20250815 01:52:46.815030 427 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250815 01:52:46.819185 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 100a2f6a32f7408886e9b6a5cabdc213 to finish bootstrapping
W20250815 01:52:46.822937 930 tablet.cc:2378] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:52:46.833683 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f046314d04e3419db0bc3c06aae3b292 to finish bootstrapping
I20250815 01:52:46.845077 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver dc882f5ec3c742dcb8b2b84270a388f4 to finish bootstrapping
I20250815 01:52:46.857059 427 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250815 01:52:46.860786 427 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "7d08a1abafe2478ab98e528d66854315"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "f046314d04e3419db0bc3c06aae3b292"
rpc_addresses {
host: "127.0.106.194"
port: 42751
}
}
ts_infos {
permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213"
rpc_addresses {
host: "127.0.106.193"
port: 34581
}
}
ts_infos {
permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
rpc_addresses {
host: "127.0.106.195"
port: 42383
}
}
I20250815 01:52:47.263571 1090 consensus_queue.cc:1035] T 7d08a1abafe2478ab98e528d66854315 P 100a2f6a32f7408886e9b6a5cabdc213 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:52:47.279575 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 533
W20250815 01:52:47.309389 686 connection.cc:537] server connection from 127.0.106.193:59057 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250815 01:52:47.309950 460 connection.cc:537] server connection from 127.0.106.193:53775 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250815 01:52:47.310549 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 441
I20250815 01:52:47.341259 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:39275
--webserver_interface=127.0.106.254
--webserver_port=40745
--builtin_ntp_servers=127.0.106.212:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:39275 with env {}
I20250815 01:52:47.468986 1063 heartbeater.cc:499] Master 127.0.106.254:39275 was elected leader, sending a full tablet report...
W20250815 01:52:47.472975 1063 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:39275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:39275: connect: Connection refused (error 111)
W20250815 01:52:47.658028 1103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:47.658664 1103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:52:47.659085 1103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:47.691151 1103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:52:47.691449 1103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:52:47.691658 1103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:52:47.691905 1103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20250815 01:52:47.717061 929 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:39275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:39275: connect: Connection refused (error 111)
I20250815 01:52:47.731194 1103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:39275
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=40745
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:52:47.732472 1103 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:52:47.734036 1103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:52:47.744648 1111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:52:48.319402 1118 raft_consensus.cc:491] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:52:48.320067 1118 raft_consensus.cc:513] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
W20250815 01:52:48.325479 795 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:39275 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:39275: connect: Connection refused (error 111)
I20250815 01:52:48.339534 1118 leader_election.cc:290] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f046314d04e3419db0bc3c06aae3b292 (127.0.106.194:42751), 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581)
W20250815 01:52:48.353120 816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.193:34581: connect: Connection refused (error 111)
W20250815 01:52:48.371297 816 leader_election.cc:336] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581): Network error: Client connection negotiation failed: client connection to 127.0.106.193:34581: connect: Connection refused (error 111)
I20250815 01:52:48.371229 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f046314d04e3419db0bc3c06aae3b292" is_pre_election: true
I20250815 01:52:48.373548 816 leader_election.cc:304] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc882f5ec3c742dcb8b2b84270a388f4; no voters: 100a2f6a32f7408886e9b6a5cabdc213, f046314d04e3419db0bc3c06aae3b292
I20250815 01:52:48.374930 1118 raft_consensus.cc:2747] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250815 01:52:48.795794 1124 raft_consensus.cc:491] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 100a2f6a32f7408886e9b6a5cabdc213)
I20250815 01:52:48.796545 1124 raft_consensus.cc:513] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
W20250815 01:52:48.803027 683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.193:34581: connect: Connection refused (error 111)
I20250815 01:52:48.812520 1124 leader_election.cc:290] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581), dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195:42383)
W20250815 01:52:48.828429 683 leader_election.cc:336] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581): Network error: Client connection negotiation failed: client connection to 127.0.106.193:34581: connect: Connection refused (error 111)
I20250815 01:52:48.833076 884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "f046314d04e3419db0bc3c06aae3b292" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" is_pre_election: true
I20250815 01:52:48.833904 884 raft_consensus.cc:2466] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f046314d04e3419db0bc3c06aae3b292 in term 1.
I20250815 01:52:48.835681 686 leader_election.cc:304] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc882f5ec3c742dcb8b2b84270a388f4, f046314d04e3419db0bc3c06aae3b292; no voters: 100a2f6a32f7408886e9b6a5cabdc213
I20250815 01:52:48.837218 1124 raft_consensus.cc:2802] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 01:52:48.837651 1124 raft_consensus.cc:491] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 100a2f6a32f7408886e9b6a5cabdc213)
I20250815 01:52:48.838070 1124 raft_consensus.cc:3058] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:52:48.847734 1124 raft_consensus.cc:513] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
W20250815 01:52:48.856051 683 leader_election.cc:336] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581): Network error: Client connection negotiation failed: client connection to 127.0.106.193:34581: connect: Connection refused (error 111)
I20250815 01:52:48.858887 884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "f046314d04e3419db0bc3c06aae3b292" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
I20250815 01:52:48.859597 884 raft_consensus.cc:3058] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:52:48.868592 884 raft_consensus.cc:2466] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f046314d04e3419db0bc3c06aae3b292 in term 2.
I20250815 01:52:48.869232 1124 leader_election.cc:290] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 2 election: Requested vote from peers 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581), dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195:42383)
I20250815 01:52:48.871153 686 leader_election.cc:304] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc882f5ec3c742dcb8b2b84270a388f4, f046314d04e3419db0bc3c06aae3b292; no voters: 100a2f6a32f7408886e9b6a5cabdc213
I20250815 01:52:48.872655 1124 raft_consensus.cc:2802] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 01:52:48.878010 1124 raft_consensus.cc:695] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 2 LEADER]: Becoming Leader. State: Replica: f046314d04e3419db0bc3c06aae3b292, State: Running, Role: LEADER
I20250815 01:52:48.879360 1124 consensus_queue.cc:237] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
W20250815 01:52:49.151826 1110 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 1103
W20250815 01:52:49.255702 1110 kernel_stack_watchdog.cc:198] Thread 1103 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 403ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:52:47.745132 1112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:49.257157 1103 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.512s user 0.504s sys 0.969s
W20250815 01:52:49.257551 1103 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.513s user 0.504s sys 0.970s
W20250815 01:52:49.257997 1113 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1510 milliseconds
W20250815 01:52:49.259464 1114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:52:49.259405 1103 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:52:49.262578 1103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:52:49.264994 1103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:52:49.266371 1103 hybrid_clock.cc:648] HybridClock initialized: now 1755222769266303 us; error 41 us; skew 500 ppm
I20250815 01:52:49.267288 1103 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:52:49.273313 1103 webserver.cc:480] Webserver started at http://127.0.106.254:40745/ using document root <none> and password file <none>
I20250815 01:52:49.274209 1103 fs_manager.cc:362] Metadata directory not provided
I20250815 01:52:49.274468 1103 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:52:49.282135 1103 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20250815 01:52:49.286615 1134 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:49.287689 1103 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 01:52:49.288012 1103 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5603f3984aec4573b3407256b696508f"
format_stamp: "Formatted at 2025-08-15 01:52:37 on dist-test-slave-h491"
I20250815 01:52:49.289889 1103 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250815 01:52:49.330929 683 consensus_peers.cc:489] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 -> Peer 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581): Couldn't send request to peer 100a2f6a32f7408886e9b6a5cabdc213. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.193:34581: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 01:52:49.342010 884 raft_consensus.cc:1273] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 2 FOLLOWER]: Refusing update from remote peer f046314d04e3419db0bc3c06aae3b292: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 01:52:49.343216 1124 consensus_queue.cc:1035] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:52:49.344159 1103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:52:49.345757 1103 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:52:49.346194 1103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:52:49.419381 1103 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:39275
I20250815 01:52:49.419464 1190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:39275 every 8 connection(s)
I20250815 01:52:49.422302 1103 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:52:49.432451 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 1103
I20250815 01:52:49.432602 1191 sys_catalog.cc:263] Verifying existing consensus state
I20250815 01:52:49.433079 427 kudu-admin-test.cc:735] Forcing unsafe config change on tserver f046314d04e3419db0bc3c06aae3b292
I20250815 01:52:49.440853 1191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Bootstrap starting.
I20250815 01:52:49.479040 1191 log.cc:826] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Log is configured to *not* fsync() on all Append() calls
I20250815 01:52:49.508080 1191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:52:49.509817 1191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Bootstrap complete.
I20250815 01:52:49.530763 1063 heartbeater.cc:344] Connected to a master server at 127.0.106.254:39275
I20250815 01:52:49.531296 1063 heartbeater.cc:499] Master 127.0.106.254:39275 was elected leader, sending a full tablet report...
I20250815 01:52:49.536293 1191 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } }
I20250815 01:52:49.538836 1191 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5603f3984aec4573b3407256b696508f, State: Initialized, Role: FOLLOWER
I20250815 01:52:49.539613 1191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } }
I20250815 01:52:49.540107 1191 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:52:49.540380 1191 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:52:49.540704 1191 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:52:49.547225 1191 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } }
I20250815 01:52:49.547971 1191 leader_election.cc:304] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5603f3984aec4573b3407256b696508f; no voters:
I20250815 01:52:49.550208 1191 leader_election.cc:290] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 01:52:49.550539 1197 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 2 FOLLOWER]: Leader election won for term 2
I20250815 01:52:49.553879 1197 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [term 2 LEADER]: Becoming Leader. State: Replica: 5603f3984aec4573b3407256b696508f, State: Running, Role: LEADER
I20250815 01:52:49.554813 1197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } }
I20250815 01:52:49.555735 1191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:52:49.566089 1198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5603f3984aec4573b3407256b696508f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } } }
I20250815 01:52:49.566978 1198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: This master's current role is: LEADER
I20250815 01:52:49.569413 1199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5603f3984aec4573b3407256b696508f. Latest consensus state: current_term: 2 leader_uuid: "5603f3984aec4573b3407256b696508f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5603f3984aec4573b3407256b696508f" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 39275 } } }
I20250815 01:52:49.570246 1199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f [sys.catalog]: This master's current role is: LEADER
I20250815 01:52:49.576932 1204 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:52:49.601719 1204 catalog_manager.cc:671] Loaded metadata for table TestTable [id=33cc433e1f8241e290bf1021de91287a]
I20250815 01:52:49.611455 1204 tablet_loader.cc:96] loaded metadata for tablet 7d08a1abafe2478ab98e528d66854315 (table TestTable [id=33cc433e1f8241e290bf1021de91287a])
I20250815 01:52:49.612897 1204 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:52:49.617617 1204 catalog_manager.cc:1261] Loaded cluster ID: d8ecc26b485f42e4b33fa89c6233f6d6
I20250815 01:52:49.618016 1204 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:52:49.626521 1204 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:52:49.631132 1204 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5603f3984aec4573b3407256b696508f: Loaded TSK: 0
I20250815 01:52:49.632750 1204 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 01:52:49.838248 1193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:49.838904 1193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:49.871987 1193 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250815 01:52:50.403167 795 heartbeater.cc:344] Connected to a master server at 127.0.106.254:39275
I20250815 01:52:50.412132 1156 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" instance_seqno: 1755222762664931) as {username='slave'} at 127.0.106.194:53161; Asking this server to re-register.
I20250815 01:52:50.414054 795 heartbeater.cc:461] Registering TS with master...
I20250815 01:52:50.414726 795 heartbeater.cc:507] Master 127.0.106.254:39275 requested a full tablet report, sending...
I20250815 01:52:50.420156 1156 ts_manager.cc:194] Registered new tserver with Master: f046314d04e3419db0bc3c06aae3b292 (127.0.106.194:42751)
I20250815 01:52:50.422489 929 heartbeater.cc:344] Connected to a master server at 127.0.106.254:39275
I20250815 01:52:50.425611 1155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" instance_seqno: 1755222764524110) as {username='slave'} at 127.0.106.195:37173; Asking this server to re-register.
I20250815 01:52:50.427048 929 heartbeater.cc:461] Registering TS with master...
I20250815 01:52:50.427659 929 heartbeater.cc:507] Master 127.0.106.254:39275 requested a full tablet report, sending...
I20250815 01:52:50.431798 1156 catalog_manager.cc:5582] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 reported cstate change: term changed from 1 to 2, leader changed from 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193) to f046314d04e3419db0bc3c06aae3b292 (127.0.106.194). New cstate: current_term: 2 leader_uuid: "f046314d04e3419db0bc3c06aae3b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } health_report { overall_health: HEALTHY } } }
I20250815 01:52:50.439599 1155 ts_manager.cc:194] Registered new tserver with Master: dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195:42383)
I20250815 01:52:50.535058 1063 heartbeater.cc:499] Master 127.0.106.254:39275 was elected leader, sending a full tablet report...
I20250815 01:52:50.546679 1154 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4884b647203a42c18c5875acba819b78" instance_seqno: 1755222766394112) as {username='slave'} at 127.0.106.196:59377; Asking this server to re-register.
I20250815 01:52:50.548709 1063 heartbeater.cc:461] Registering TS with master...
I20250815 01:52:50.549374 1063 heartbeater.cc:507] Master 127.0.106.254:39275 requested a full tablet report, sending...
I20250815 01:52:50.561120 1154 ts_manager.cc:194] Registered new tserver with Master: 4884b647203a42c18c5875acba819b78 (127.0.106.196:35561)
W20250815 01:52:51.318099 1222 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 1193
W20250815 01:52:51.318861 1222 kernel_stack_watchdog.cc:198] Thread 1193 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:52:51.491562 1193 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.576s user 0.603s sys 0.948s
W20250815 01:52:51.631108 1193 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.716s user 0.612s sys 0.953s
I20250815 01:52:51.684386 750 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "f046314d04e3419db0bc3c06aae3b292"
tablet_id: "7d08a1abafe2478ab98e528d66854315"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
}
peers {
permanent_uuid: "f046314d04e3419db0bc3c06aae3b292"
}
}
from {username='slave'} at 127.0.0.1:51236
W20250815 01:52:51.685619 750 raft_consensus.cc:2216] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } unsafe_config_change: true
I20250815 01:52:51.686571 750 raft_consensus.cc:3053] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 2 LEADER]: Stepping down as leader of term 2
I20250815 01:52:51.686786 750 raft_consensus.cc:738] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f046314d04e3419db0bc3c06aae3b292, State: Running, Role: LEADER
I20250815 01:52:51.687342 750 consensus_queue.cc:260] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:51.688262 750 raft_consensus.cc:3058] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 2 FOLLOWER]: Advancing to term 3
I20250815 01:52:53.061826 1249 raft_consensus.cc:491] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f046314d04e3419db0bc3c06aae3b292)
I20250815 01:52:53.062538 1249 raft_consensus.cc:513] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } }
I20250815 01:52:53.064339 1249 leader_election.cc:290] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f046314d04e3419db0bc3c06aae3b292 (127.0.106.194:42751), 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581)
I20250815 01:52:53.065308 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f046314d04e3419db0bc3c06aae3b292" is_pre_election: true
W20250815 01:52:53.069901 816 leader_election.cc:336] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193:34581): Network error: Client connection negotiation failed: client connection to 127.0.106.193:34581: connect: Connection refused (error 111)
I20250815 01:52:53.070223 816 leader_election.cc:304] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc882f5ec3c742dcb8b2b84270a388f4; no voters: 100a2f6a32f7408886e9b6a5cabdc213, f046314d04e3419db0bc3c06aae3b292
I20250815 01:52:53.070827 1249 raft_consensus.cc:2747] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250815 01:52:53.194425 1252 raft_consensus.cc:491] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250815 01:52:53.194809 1252 raft_consensus.cc:513] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } unsafe_config_change: true
I20250815 01:52:53.195863 1252 leader_election.cc:290] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195:42383)
I20250815 01:52:53.197261 884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "f046314d04e3419db0bc3c06aae3b292" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" is_pre_election: true
I20250815 01:52:53.197784 884 raft_consensus.cc:2466] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f046314d04e3419db0bc3c06aae3b292 in term 2.
I20250815 01:52:53.198809 686 leader_election.cc:304] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: dc882f5ec3c742dcb8b2b84270a388f4, f046314d04e3419db0bc3c06aae3b292; no voters:
I20250815 01:52:53.199386 1252 raft_consensus.cc:2802] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250815 01:52:53.199630 1252 raft_consensus.cc:491] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250815 01:52:53.199839 1252 raft_consensus.cc:3058] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 3 FOLLOWER]: Advancing to term 4
I20250815 01:52:53.204025 1252 raft_consensus.cc:513] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } unsafe_config_change: true
I20250815 01:52:53.204988 1252 leader_election.cc:290] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 4 election: Requested vote from peers dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195:42383)
I20250815 01:52:53.205976 884 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d08a1abafe2478ab98e528d66854315" candidate_uuid: "f046314d04e3419db0bc3c06aae3b292" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
I20250815 01:52:53.206429 884 raft_consensus.cc:3058] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 2 FOLLOWER]: Advancing to term 4
I20250815 01:52:53.210744 884 raft_consensus.cc:2466] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f046314d04e3419db0bc3c06aae3b292 in term 4.
I20250815 01:52:53.211594 686 leader_election.cc:304] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: dc882f5ec3c742dcb8b2b84270a388f4, f046314d04e3419db0bc3c06aae3b292; no voters:
I20250815 01:52:53.212188 1252 raft_consensus.cc:2802] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 4 FOLLOWER]: Leader election won for term 4
I20250815 01:52:53.212975 1252 raft_consensus.cc:695] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 4 LEADER]: Becoming Leader. State: Replica: f046314d04e3419db0bc3c06aae3b292, State: Running, Role: LEADER
I20250815 01:52:53.213649 1252 consensus_queue.cc:237] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } unsafe_config_change: true
I20250815 01:52:53.219660 1155 catalog_manager.cc:5582] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER f046314d04e3419db0bc3c06aae3b292 (127.0.106.194), VOTER dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195). New cstate: current_term: 4 leader_uuid: "f046314d04e3419db0bc3c06aae3b292" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100a2f6a32f7408886e9b6a5cabdc213" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34581 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } unsafe_config_change: true }
I20250815 01:52:53.610981 884 raft_consensus.cc:1273] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 4 FOLLOWER]: Refusing update from remote peer f046314d04e3419db0bc3c06aae3b292: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250815 01:52:53.612181 1252 consensus_queue.cc:1035] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250815 01:52:53.619953 1253 raft_consensus.cc:2953] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } unsafe_config_change: true }
I20250815 01:52:53.621320 884 raft_consensus.cc:2953] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } unsafe_config_change: true }
I20250815 01:52:53.633172 1155 catalog_manager.cc:5582] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 reported cstate change: config changed from index -1 to 3, VOTER 100a2f6a32f7408886e9b6a5cabdc213 (127.0.106.193) evicted, no longer has a pending config: VOTER f046314d04e3419db0bc3c06aae3b292 (127.0.106.194), VOTER dc882f5ec3c742dcb8b2b84270a388f4 (127.0.106.195). New cstate: current_term: 4 leader_uuid: "f046314d04e3419db0bc3c06aae3b292" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250815 01:52:53.640172 1155 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 7d08a1abafe2478ab98e528d66854315 on TS 100a2f6a32f7408886e9b6a5cabdc213: Not found: failed to reset TS proxy: Could not find TS for UUID 100a2f6a32f7408886e9b6a5cabdc213
I20250815 01:52:53.656744 750 consensus_queue.cc:237] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: NON_VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: true } } unsafe_config_change: true
I20250815 01:52:53.663380 884 raft_consensus.cc:1273] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 4 FOLLOWER]: Refusing update from remote peer f046314d04e3419db0bc3c06aae3b292: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250815 01:52:53.665045 1253 consensus_queue.cc:1035] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250815 01:52:53.672466 1252 raft_consensus.cc:2953] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 4884b647203a42c18c5875acba819b78 (127.0.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: NON_VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: true } } unsafe_config_change: true }
I20250815 01:52:53.684943 1135 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7d08a1abafe2478ab98e528d66854315 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250815 01:52:53.684466 884 raft_consensus.cc:2953] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 4884b647203a42c18c5875acba819b78 (127.0.106.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: NON_VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: true } } unsafe_config_change: true }
W20250815 01:52:53.692698 1136 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 7d08a1abafe2478ab98e528d66854315 on TS 100a2f6a32f7408886e9b6a5cabdc213 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 100a2f6a32f7408886e9b6a5cabdc213
W20250815 01:52:53.692757 684 consensus_peers.cc:489] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 -> Peer 4884b647203a42c18c5875acba819b78 (127.0.106.196:35561): Couldn't send request to peer 4884b647203a42c18c5875acba819b78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d08a1abafe2478ab98e528d66854315. This is attempt 1: this message will repeat every 5th retry.
I20250815 01:52:53.693760 1155 catalog_manager.cc:5582] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 reported cstate change: config changed from index 3 to 5, NON_VOTER 4884b647203a42c18c5875acba819b78 (127.0.106.196) added. New cstate: current_term: 4 leader_uuid: "f046314d04e3419db0bc3c06aae3b292" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: NON_VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
I20250815 01:52:54.218324 1268 ts_tablet_manager.cc:927] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: Initiating tablet copy from peer f046314d04e3419db0bc3c06aae3b292 (127.0.106.194:42751)
I20250815 01:52:54.220352 1268 tablet_copy_client.cc:323] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.194:42751
I20250815 01:52:54.230217 770 tablet_copy_service.cc:140] P f046314d04e3419db0bc3c06aae3b292: Received BeginTabletCopySession request for tablet 7d08a1abafe2478ab98e528d66854315 from peer 4884b647203a42c18c5875acba819b78 ({username='slave'} at 127.0.106.196:37579)
I20250815 01:52:54.230706 770 tablet_copy_service.cc:161] P f046314d04e3419db0bc3c06aae3b292: Beginning new tablet copy session on tablet 7d08a1abafe2478ab98e528d66854315 from peer 4884b647203a42c18c5875acba819b78 at {username='slave'} at 127.0.106.196:37579: session id = 4884b647203a42c18c5875acba819b78-7d08a1abafe2478ab98e528d66854315
I20250815 01:52:54.235380 770 tablet_copy_source_session.cc:215] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 01:52:54.240478 1268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d08a1abafe2478ab98e528d66854315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:52:54.257225 1268 tablet_copy_client.cc:806] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: tablet copy: Starting download of 0 data blocks...
I20250815 01:52:54.257828 1268 tablet_copy_client.cc:670] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: tablet copy: Starting download of 1 WAL segments...
I20250815 01:52:54.261348 1268 tablet_copy_client.cc:538] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 01:52:54.267957 1268 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: Bootstrap starting.
I20250815 01:52:54.279592 1268 log.cc:826] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: Log is configured to *not* fsync() on all Append() calls
I20250815 01:52:54.290889 1268 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:52:54.291543 1268 tablet_bootstrap.cc:492] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: Bootstrap complete.
I20250815 01:52:54.292114 1268 ts_tablet_manager.cc:1397] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.009s
I20250815 01:52:54.309384 1268 raft_consensus.cc:357] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: NON_VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: true } } unsafe_config_change: true
I20250815 01:52:54.310190 1268 raft_consensus.cc:738] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 4884b647203a42c18c5875acba819b78, State: Initialized, Role: LEARNER
I20250815 01:52:54.310725 1268 consensus_queue.cc:260] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: NON_VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: true } } unsafe_config_change: true
I20250815 01:52:54.314000 1268 ts_tablet_manager.cc:1428] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78: Time spent starting tablet: real 0.022s user 0.022s sys 0.000s
I20250815 01:52:54.315673 770 tablet_copy_service.cc:342] P f046314d04e3419db0bc3c06aae3b292: Request end of tablet copy session 4884b647203a42c18c5875acba819b78-7d08a1abafe2478ab98e528d66854315 received from {username='slave'} at 127.0.106.196:37579
I20250815 01:52:54.316093 770 tablet_copy_service.cc:434] P f046314d04e3419db0bc3c06aae3b292: ending tablet copy session 4884b647203a42c18c5875acba819b78-7d08a1abafe2478ab98e528d66854315 on tablet 7d08a1abafe2478ab98e528d66854315 with peer 4884b647203a42c18c5875acba819b78
I20250815 01:52:54.781287 1018 raft_consensus.cc:1215] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250815 01:52:54.858980 1136 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 7d08a1abafe2478ab98e528d66854315 on TS 100a2f6a32f7408886e9b6a5cabdc213 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 100a2f6a32f7408886e9b6a5cabdc213
I20250815 01:52:55.190711 1276 raft_consensus.cc:1062] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292: attempting to promote NON_VOTER 4884b647203a42c18c5875acba819b78 to VOTER
I20250815 01:52:55.192234 1276 consensus_queue.cc:237] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: false } } unsafe_config_change: true
I20250815 01:52:55.196759 884 raft_consensus.cc:1273] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 4 FOLLOWER]: Refusing update from remote peer f046314d04e3419db0bc3c06aae3b292: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250815 01:52:55.197024 1018 raft_consensus.cc:1273] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78 [term 4 LEARNER]: Refusing update from remote peer f046314d04e3419db0bc3c06aae3b292: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250815 01:52:55.197964 1275 consensus_queue.cc:1035] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250815 01:52:55.198827 1267 consensus_queue.cc:1035] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250815 01:52:55.204970 1275 raft_consensus.cc:2953] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 4884b647203a42c18c5875acba819b78 (127.0.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: false } } unsafe_config_change: true }
I20250815 01:52:55.206609 884 raft_consensus.cc:2953] T 7d08a1abafe2478ab98e528d66854315 P dc882f5ec3c742dcb8b2b84270a388f4 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 4884b647203a42c18c5875acba819b78 (127.0.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: false } } unsafe_config_change: true }
I20250815 01:52:55.208614 1018 raft_consensus.cc:2953] T 7d08a1abafe2478ab98e528d66854315 P 4884b647203a42c18c5875acba819b78 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 4884b647203a42c18c5875acba819b78 (127.0.106.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: false } } unsafe_config_change: true }
I20250815 01:52:55.217227 1155 catalog_manager.cc:5582] T 7d08a1abafe2478ab98e528d66854315 P f046314d04e3419db0bc3c06aae3b292 reported cstate change: config changed from index 5 to 6, 4884b647203a42c18c5875acba819b78 (127.0.106.196) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "f046314d04e3419db0bc3c06aae3b292" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f046314d04e3419db0bc3c06aae3b292" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 42751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4884b647203a42c18c5875acba819b78" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 35561 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250815 01:52:55.224241 427 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250815 01:52:55.244007 427 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "7d08a1abafe2478ab98e528d66854315"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "f046314d04e3419db0bc3c06aae3b292"
rpc_addresses {
host: "127.0.106.194"
port: 42751
}
}
ts_infos {
permanent_uuid: "dc882f5ec3c742dcb8b2b84270a388f4"
rpc_addresses {
host: "127.0.106.195"
port: 42383
}
}
ts_infos {
permanent_uuid: "4884b647203a42c18c5875acba819b78"
rpc_addresses {
host: "127.0.106.196"
port: 35561
}
}
I20250815 01:52:55.248250 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 666
I20250815 01:52:55.275101 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 799
I20250815 01:52:55.302307 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 933
I20250815 01:52:55.327029 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 1103
2025-08-15T01:52:55Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (20940 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250815 01:52:55.390676 427 test_util.cc:276] Using random seed: -1509465672
I20250815 01:52:55.396626 427 ts_itest-base.cc:115] Starting cluster with:
I20250815 01:52:55.396809 427 ts_itest-base.cc:116] --------------
I20250815 01:52:55.396979 427 ts_itest-base.cc:117] 3 tablet servers
I20250815 01:52:55.397153 427 ts_itest-base.cc:118] 3 replicas per TS
I20250815 01:52:55.397298 427 ts_itest-base.cc:119] --------------
2025-08-15T01:52:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T01:52:55Z Disabled control of system clock
I20250815 01:52:55.434314 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:33557
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:33557
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250815 01:52:55.732759 1295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:55.733340 1295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:52:55.733765 1295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:55.765609 1295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:52:55.765918 1295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:52:55.766127 1295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:52:55.766325 1295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:52:55.802093 1295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:33557
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:52:55.803371 1295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:52:55.805078 1295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:52:55.816114 1301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:57.220036 1300 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 1295
W20250815 01:52:57.619513 1295 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.803s user 0.654s sys 1.149s
W20250815 01:52:55.816161 1302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:57.619906 1295 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.804s user 0.654s sys 1.149s
W20250815 01:52:57.622399 1304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:52:57.626263 1303 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1808 milliseconds
I20250815 01:52:57.626379 1295 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:52:57.627914 1295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:52:57.630306 1295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:52:57.631755 1295 hybrid_clock.cc:648] HybridClock initialized: now 1755222777631699 us; error 69 us; skew 500 ppm
I20250815 01:52:57.632575 1295 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:52:57.638566 1295 webserver.cc:480] Webserver started at http://127.0.106.254:35659/ using document root <none> and password file <none>
I20250815 01:52:57.639484 1295 fs_manager.cc:362] Metadata directory not provided
I20250815 01:52:57.639683 1295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:52:57.640156 1295 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:52:57.644503 1295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "69bf76b2e86044e6bb5ed84ed61c4485"
format_stamp: "Formatted at 2025-08-15 01:52:57 on dist-test-slave-h491"
I20250815 01:52:57.645596 1295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "69bf76b2e86044e6bb5ed84ed61c4485"
format_stamp: "Formatted at 2025-08-15 01:52:57 on dist-test-slave-h491"
I20250815 01:52:57.652802 1295 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250815 01:52:57.658375 1311 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:52:57.659438 1295 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250815 01:52:57.659778 1295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "69bf76b2e86044e6bb5ed84ed61c4485"
format_stamp: "Formatted at 2025-08-15 01:52:57 on dist-test-slave-h491"
I20250815 01:52:57.660100 1295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:52:57.706820 1295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:52:57.708249 1295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:52:57.708674 1295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:52:57.779256 1295 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:33557
I20250815 01:52:57.779433 1362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:33557 every 8 connection(s)
I20250815 01:52:57.781940 1295 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:52:57.787099 1363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:52:57.787532 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 1295
I20250815 01:52:57.787884 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 01:52:57.811280 1363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485: Bootstrap starting.
I20250815 01:52:57.817848 1363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485: Neither blocks nor log segments found. Creating new log.
I20250815 01:52:57.819581 1363 log.cc:826] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485: Log is configured to *not* fsync() on all Append() calls
I20250815 01:52:57.824092 1363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485: No bootstrap required, opened a new log
I20250815 01:52:57.841408 1363 raft_consensus.cc:357] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf76b2e86044e6bb5ed84ed61c4485" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 33557 } }
I20250815 01:52:57.842123 1363 raft_consensus.cc:383] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:52:57.842420 1363 raft_consensus.cc:738] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69bf76b2e86044e6bb5ed84ed61c4485, State: Initialized, Role: FOLLOWER
I20250815 01:52:57.843149 1363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf76b2e86044e6bb5ed84ed61c4485" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 33557 } }
I20250815 01:52:57.843636 1363 raft_consensus.cc:397] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:52:57.843930 1363 raft_consensus.cc:491] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:52:57.844247 1363 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:52:57.848641 1363 raft_consensus.cc:513] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf76b2e86044e6bb5ed84ed61c4485" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 33557 } }
I20250815 01:52:57.849334 1363 leader_election.cc:304] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69bf76b2e86044e6bb5ed84ed61c4485; no voters:
I20250815 01:52:57.851001 1363 leader_election.cc:290] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:52:57.851758 1368 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:52:57.853897 1368 raft_consensus.cc:695] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [term 1 LEADER]: Becoming Leader. State: Replica: 69bf76b2e86044e6bb5ed84ed61c4485, State: Running, Role: LEADER
I20250815 01:52:57.854835 1368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf76b2e86044e6bb5ed84ed61c4485" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 33557 } }
I20250815 01:52:57.856739 1363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:52:57.863870 1370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69bf76b2e86044e6bb5ed84ed61c4485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf76b2e86044e6bb5ed84ed61c4485" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 33557 } } }
I20250815 01:52:57.864699 1370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [sys.catalog]: This master's current role is: LEADER
I20250815 01:52:57.865067 1369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69bf76b2e86044e6bb5ed84ed61c4485. Latest consensus state: current_term: 1 leader_uuid: "69bf76b2e86044e6bb5ed84ed61c4485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69bf76b2e86044e6bb5ed84ed61c4485" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 33557 } } }
I20250815 01:52:57.865679 1369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485 [sys.catalog]: This master's current role is: LEADER
I20250815 01:52:57.868374 1376 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:52:57.878916 1376 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:52:57.896239 1376 catalog_manager.cc:1349] Generated new cluster ID: 1edb4d6a0827437398c539df443425e2
I20250815 01:52:57.896574 1376 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:52:57.935223 1376 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:52:57.936786 1376 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:52:57.954715 1376 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 69bf76b2e86044e6bb5ed84ed61c4485: Generated new TSK 0
I20250815 01:52:57.955907 1376 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 01:52:57.983632 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:33557
--builtin_ntp_servers=127.0.106.212:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250815 01:52:58.302475 1387 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 01:52:58.303143 1387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:52:58.303409 1387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:52:58.303885 1387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:52:58.335888 1387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:52:58.336779 1387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:52:58.372802 1387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:52:58.374214 1387 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:52:58.375875 1387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:52:58.390892 1394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:00.161458 1395 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1770 milliseconds
W20250815 01:53:00.161352 1387 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.770s user 0.002s sys 0.000s
W20250815 01:53:00.162109 1387 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.771s user 0.002s sys 0.000s
W20250815 01:52:59.790215 1392 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 1387
W20250815 01:52:58.391029 1393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:00.163614 1387 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 01:53:00.163678 1397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:00.167224 1387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:00.173861 1387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:00.175315 1387 hybrid_clock.cc:648] HybridClock initialized: now 1755222780175276 us; error 59 us; skew 500 ppm
I20250815 01:53:00.176141 1387 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:00.182657 1387 webserver.cc:480] Webserver started at http://127.0.106.193:44419/ using document root <none> and password file <none>
I20250815 01:53:00.183660 1387 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:00.183887 1387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:00.184319 1387 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:00.188694 1387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "366a46a9ba144f6c8e75e58fdd32e83e"
format_stamp: "Formatted at 2025-08-15 01:53:00 on dist-test-slave-h491"
I20250815 01:53:00.189786 1387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "366a46a9ba144f6c8e75e58fdd32e83e"
format_stamp: "Formatted at 2025-08-15 01:53:00 on dist-test-slave-h491"
I20250815 01:53:00.197190 1387 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250815 01:53:00.203440 1403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:00.204609 1387 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250815 01:53:00.204926 1387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "366a46a9ba144f6c8e75e58fdd32e83e"
format_stamp: "Formatted at 2025-08-15 01:53:00 on dist-test-slave-h491"
I20250815 01:53:00.205245 1387 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:00.277740 1387 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:00.279249 1387 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:00.279711 1387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:00.282294 1387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:00.287122 1387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:00.287416 1387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:00.287703 1387 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:00.287933 1387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:00.456442 1387 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:33153
I20250815 01:53:00.456563 1515 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:33153 every 8 connection(s)
I20250815 01:53:00.459185 1387 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:53:00.459949 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 1387
I20250815 01:53:00.460376 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 01:53:00.467886 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:33557
--builtin_ntp_servers=127.0.106.212:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250815 01:53:00.491130 1516 heartbeater.cc:344] Connected to a master server at 127.0.106.254:33557
I20250815 01:53:00.491596 1516 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:00.492753 1516 heartbeater.cc:507] Master 127.0.106.254:33557 requested a full tablet report, sending...
I20250815 01:53:00.495307 1328 ts_manager.cc:194] Registered new tserver with Master: 366a46a9ba144f6c8e75e58fdd32e83e (127.0.106.193:33153)
I20250815 01:53:00.497205 1328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:60127
W20250815 01:53:00.795207 1520 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 01:53:00.795847 1520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:00.796111 1520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:00.796593 1520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:00.828642 1520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:00.829519 1520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:53:00.865491 1520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:00.866880 1520 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:00.868439 1520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:00.880148 1526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:01.500520 1516 heartbeater.cc:499] Master 127.0.106.254:33557 was elected leader, sending a full tablet report...
W20250815 01:53:02.284615 1525 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 1520
W20250815 01:53:02.618153 1520 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.738s user 0.685s sys 1.052s
W20250815 01:53:00.880682 1527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:02.618562 1520 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.738s user 0.686s sys 1.052s
W20250815 01:53:02.621158 1529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:02.623009 1528 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1738 milliseconds
I20250815 01:53:02.623113 1520 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:02.624317 1520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:02.626488 1520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:02.627883 1520 hybrid_clock.cc:648] HybridClock initialized: now 1755222782627831 us; error 65 us; skew 500 ppm
I20250815 01:53:02.628664 1520 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:02.634953 1520 webserver.cc:480] Webserver started at http://127.0.106.194:33189/ using document root <none> and password file <none>
I20250815 01:53:02.635928 1520 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:02.636158 1520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:02.636611 1520 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:02.641242 1520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b48ee44b093f4326a21d91a46610b10c"
format_stamp: "Formatted at 2025-08-15 01:53:02 on dist-test-slave-h491"
I20250815 01:53:02.642458 1520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b48ee44b093f4326a21d91a46610b10c"
format_stamp: "Formatted at 2025-08-15 01:53:02 on dist-test-slave-h491"
I20250815 01:53:02.649928 1520 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20250815 01:53:02.655982 1536 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:02.657255 1520 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250815 01:53:02.657611 1520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b48ee44b093f4326a21d91a46610b10c"
format_stamp: "Formatted at 2025-08-15 01:53:02 on dist-test-slave-h491"
I20250815 01:53:02.657975 1520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:02.723619 1520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:02.725128 1520 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:02.725538 1520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:02.728156 1520 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:02.732309 1520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:02.732565 1520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:02.732769 1520 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:02.732916 1520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:02.874228 1520 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:46391
I20250815 01:53:02.874377 1648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:46391 every 8 connection(s)
I20250815 01:53:02.876976 1520 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 01:53:02.879949 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 1520
I20250815 01:53:02.880532 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 01:53:02.888654 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:33557
--builtin_ntp_servers=127.0.106.212:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250815 01:53:02.905959 1649 heartbeater.cc:344] Connected to a master server at 127.0.106.254:33557
I20250815 01:53:02.906467 1649 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:02.907528 1649 heartbeater.cc:507] Master 127.0.106.254:33557 requested a full tablet report, sending...
I20250815 01:53:02.909860 1328 ts_manager.cc:194] Registered new tserver with Master: b48ee44b093f4326a21d91a46610b10c (127.0.106.194:46391)
I20250815 01:53:02.911213 1328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:52803
W20250815 01:53:03.219419 1653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 01:53:03.220083 1653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:03.220345 1653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:03.220829 1653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:03.253908 1653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:03.255004 1653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:53:03.293325 1653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:03.294778 1653 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:03.296681 1653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:03.310901 1659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:03.914657 1649 heartbeater.cc:499] Master 127.0.106.254:33557 was elected leader, sending a full tablet report...
W20250815 01:53:04.715312 1658 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 1653
W20250815 01:53:03.312286 1660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:05.084491 1653 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.773s user 0.635s sys 1.120s
W20250815 01:53:05.086241 1653 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.775s user 0.636s sys 1.120s
W20250815 01:53:05.087225 1662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:05.087191 1661 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1774 milliseconds
I20250815 01:53:05.087627 1653 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:05.092044 1653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:05.094436 1653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:05.096077 1653 hybrid_clock.cc:648] HybridClock initialized: now 1755222785096023 us; error 51 us; skew 500 ppm
I20250815 01:53:05.097270 1653 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:05.104274 1653 webserver.cc:480] Webserver started at http://127.0.106.195:42437/ using document root <none> and password file <none>
I20250815 01:53:05.105319 1653 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:05.105528 1653 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:05.106092 1653 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:05.110810 1653 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6f096cccbfe44c819b2752874360b680"
format_stamp: "Formatted at 2025-08-15 01:53:05 on dist-test-slave-h491"
I20250815 01:53:05.111989 1653 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6f096cccbfe44c819b2752874360b680"
format_stamp: "Formatted at 2025-08-15 01:53:05 on dist-test-slave-h491"
I20250815 01:53:05.119598 1653 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250815 01:53:05.125542 1670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:05.126830 1653 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 01:53:05.127193 1653 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6f096cccbfe44c819b2752874360b680"
format_stamp: "Formatted at 2025-08-15 01:53:05 on dist-test-slave-h491"
I20250815 01:53:05.127559 1653 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:05.188127 1653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:05.189635 1653 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:05.190083 1653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:05.192682 1653 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:05.197062 1653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:05.197307 1653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:05.197569 1653 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:05.197734 1653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:05.336660 1653 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:42245
I20250815 01:53:05.336757 1782 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:42245 every 8 connection(s)
I20250815 01:53:05.339300 1653 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 01:53:05.341256 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 1653
I20250815 01:53:05.341958 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 01:53:05.361101 1783 heartbeater.cc:344] Connected to a master server at 127.0.106.254:33557
I20250815 01:53:05.361583 1783 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:05.362854 1783 heartbeater.cc:507] Master 127.0.106.254:33557 requested a full tablet report, sending...
I20250815 01:53:05.365065 1328 ts_manager.cc:194] Registered new tserver with Master: 6f096cccbfe44c819b2752874360b680 (127.0.106.195:42245)
I20250815 01:53:05.366523 1328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:42467
I20250815 01:53:05.378554 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:53:05.416630 1328 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60672:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 01:53:05.435014 1328 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 01:53:05.489708 1451 tablet_service.cc:1468] Processing CreateTablet for tablet 1f45cdbbc91a4482b4740b024c0ecbde (DEFAULT_TABLE table=TestTable [id=778dbfd1c5484dd18e7b9618a1180773]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:05.490134 1584 tablet_service.cc:1468] Processing CreateTablet for tablet 1f45cdbbc91a4482b4740b024c0ecbde (DEFAULT_TABLE table=TestTable [id=778dbfd1c5484dd18e7b9618a1180773]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:05.491779 1451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f45cdbbc91a4482b4740b024c0ecbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:05.491859 1584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f45cdbbc91a4482b4740b024c0ecbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:05.491979 1718 tablet_service.cc:1468] Processing CreateTablet for tablet 1f45cdbbc91a4482b4740b024c0ecbde (DEFAULT_TABLE table=TestTable [id=778dbfd1c5484dd18e7b9618a1180773]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:05.494091 1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f45cdbbc91a4482b4740b024c0ecbde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:05.518955 1802 tablet_bootstrap.cc:492] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680: Bootstrap starting.
I20250815 01:53:05.521757 1804 tablet_bootstrap.cc:492] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e: Bootstrap starting.
I20250815 01:53:05.527875 1803 tablet_bootstrap.cc:492] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c: Bootstrap starting.
I20250815 01:53:05.529568 1804 tablet_bootstrap.cc:654] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:05.531423 1804 log.cc:826] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:05.534010 1802 tablet_bootstrap.cc:654] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:05.535789 1803 tablet_bootstrap.cc:654] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:05.536531 1802 log.cc:826] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:05.536949 1804 tablet_bootstrap.cc:492] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e: No bootstrap required, opened a new log
I20250815 01:53:05.537575 1804 ts_tablet_manager.cc:1397] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20250815 01:53:05.538298 1803 log.cc:826] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:05.542484 1802 tablet_bootstrap.cc:492] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680: No bootstrap required, opened a new log
I20250815 01:53:05.543040 1802 ts_tablet_manager.cc:1397] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.005s
I20250815 01:53:05.544310 1803 tablet_bootstrap.cc:492] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c: No bootstrap required, opened a new log
I20250815 01:53:05.544873 1803 ts_tablet_manager.cc:1397] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20250815 01:53:05.557752 1804 raft_consensus.cc:357] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:05.558735 1804 raft_consensus.cc:738] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366a46a9ba144f6c8e75e58fdd32e83e, State: Initialized, Role: FOLLOWER
I20250815 01:53:05.559473 1804 consensus_queue.cc:260] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:05.564622 1804 ts_tablet_manager.cc:1428] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e: Time spent starting tablet: real 0.027s user 0.022s sys 0.003s
I20250815 01:53:05.571317 1802 raft_consensus.cc:357] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:05.572302 1802 raft_consensus.cc:738] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f096cccbfe44c819b2752874360b680, State: Initialized, Role: FOLLOWER
I20250815 01:53:05.573117 1802 consensus_queue.cc:260] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:05.574074 1803 raft_consensus.cc:357] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:05.575225 1803 raft_consensus.cc:738] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48ee44b093f4326a21d91a46610b10c, State: Initialized, Role: FOLLOWER
I20250815 01:53:05.576262 1803 consensus_queue.cc:260] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:05.582860 1783 heartbeater.cc:499] Master 127.0.106.254:33557 was elected leader, sending a full tablet report...
I20250815 01:53:05.584738 1802 ts_tablet_manager.cc:1428] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680: Time spent starting tablet: real 0.041s user 0.035s sys 0.004s
I20250815 01:53:05.587199 1803 ts_tablet_manager.cc:1428] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c: Time spent starting tablet: real 0.042s user 0.035s sys 0.007s
I20250815 01:53:05.588716 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:53:05.592875 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 366a46a9ba144f6c8e75e58fdd32e83e to finish bootstrapping
W20250815 01:53:05.593928 1784 tablet.cc:2378] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:05.608194 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b48ee44b093f4326a21d91a46610b10c to finish bootstrapping
I20250815 01:53:05.619130 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6f096cccbfe44c819b2752874360b680 to finish bootstrapping
W20250815 01:53:05.634876 1650 tablet.cc:2378] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:05.665035 1471 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1f45cdbbc91a4482b4740b024c0ecbde"
dest_uuid: "366a46a9ba144f6c8e75e58fdd32e83e"
from {username='slave'} at 127.0.0.1:43222
I20250815 01:53:05.665665 1471 raft_consensus.cc:491] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250815 01:53:05.665970 1471 raft_consensus.cc:3058] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:05.671125 1471 raft_consensus.cc:513] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:05.673460 1471 leader_election.cc:290] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [CANDIDATE]: Term 1 election: Requested vote from peers 6f096cccbfe44c819b2752874360b680 (127.0.106.195:42245), b48ee44b093f4326a21d91a46610b10c (127.0.106.194:46391)
I20250815 01:53:05.682947 427 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250815 01:53:05.685941 1738 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f45cdbbc91a4482b4740b024c0ecbde" candidate_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6f096cccbfe44c819b2752874360b680"
I20250815 01:53:05.686203 1604 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f45cdbbc91a4482b4740b024c0ecbde" candidate_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b48ee44b093f4326a21d91a46610b10c"
I20250815 01:53:05.686578 1738 raft_consensus.cc:3058] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:05.686913 1604 raft_consensus.cc:3058] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:05.691253 1738 raft_consensus.cc:2466] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366a46a9ba144f6c8e75e58fdd32e83e in term 1.
I20250815 01:53:05.691861 1604 raft_consensus.cc:2466] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366a46a9ba144f6c8e75e58fdd32e83e in term 1.
I20250815 01:53:05.692570 1407 leader_election.cc:304] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366a46a9ba144f6c8e75e58fdd32e83e, 6f096cccbfe44c819b2752874360b680; no voters:
I20250815 01:53:05.693653 1808 raft_consensus.cc:2802] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:05.695399 1808 raft_consensus.cc:695] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 1 LEADER]: Becoming Leader. State: Replica: 366a46a9ba144f6c8e75e58fdd32e83e, State: Running, Role: LEADER
I20250815 01:53:05.696290 1808 consensus_queue.cc:237] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:05.707060 1326 catalog_manager.cc:5582] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e reported cstate change: term changed from 0 to 1, leader changed from <none> to 366a46a9ba144f6c8e75e58fdd32e83e (127.0.106.193). New cstate: current_term: 1 leader_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } health_report { overall_health: HEALTHY } } }
W20250815 01:53:05.722621 1517 tablet.cc:2378] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:05.789355 427 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250815 01:53:05.995558 427 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250815 01:53:06.099071 1808 consensus_queue.cc:1035] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:53:06.114521 1815 consensus_queue.cc:1035] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:53:08.008463 1471 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1f45cdbbc91a4482b4740b024c0ecbde"
dest_uuid: "366a46a9ba144f6c8e75e58fdd32e83e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43238
I20250815 01:53:08.009130 1471 raft_consensus.cc:604] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 1 LEADER]: Received request to transfer leadership
I20250815 01:53:08.225023 1831 raft_consensus.cc:991] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e: : Instructing follower b48ee44b093f4326a21d91a46610b10c to start an election
I20250815 01:53:08.225384 1846 raft_consensus.cc:1079] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 1 LEADER]: Signalling peer b48ee44b093f4326a21d91a46610b10c to start an election
I20250815 01:53:08.226748 1604 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1f45cdbbc91a4482b4740b024c0ecbde"
dest_uuid: "b48ee44b093f4326a21d91a46610b10c"
from {username='slave'} at 127.0.106.193:45955
I20250815 01:53:08.227226 1604 raft_consensus.cc:491] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250815 01:53:08.227458 1604 raft_consensus.cc:3058] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:53:08.231422 1604 raft_consensus.cc:513] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:08.233582 1604 leader_election.cc:290] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [CANDIDATE]: Term 2 election: Requested vote from peers 6f096cccbfe44c819b2752874360b680 (127.0.106.195:42245), 366a46a9ba144f6c8e75e58fdd32e83e (127.0.106.193:33153)
I20250815 01:53:08.244074 1738 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f45cdbbc91a4482b4740b024c0ecbde" candidate_uuid: "b48ee44b093f4326a21d91a46610b10c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6f096cccbfe44c819b2752874360b680"
I20250815 01:53:08.244542 1738 raft_consensus.cc:3058] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680 [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:53:08.245433 1471 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f45cdbbc91a4482b4740b024c0ecbde" candidate_uuid: "b48ee44b093f4326a21d91a46610b10c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "366a46a9ba144f6c8e75e58fdd32e83e"
I20250815 01:53:08.245945 1471 raft_consensus.cc:3053] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 1 LEADER]: Stepping down as leader of term 1
I20250815 01:53:08.246179 1471 raft_consensus.cc:738] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 366a46a9ba144f6c8e75e58fdd32e83e, State: Running, Role: LEADER
I20250815 01:53:08.246683 1471 consensus_queue.cc:260] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:08.247565 1471 raft_consensus.cc:3058] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:53:08.248939 1738 raft_consensus.cc:2466] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48ee44b093f4326a21d91a46610b10c in term 2.
I20250815 01:53:08.249979 1540 leader_election.cc:304] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f096cccbfe44c819b2752874360b680, b48ee44b093f4326a21d91a46610b10c; no voters:
I20250815 01:53:08.251910 1850 raft_consensus.cc:2802] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 2 FOLLOWER]: Leader election won for term 2
I20250815 01:53:08.252924 1471 raft_consensus.cc:2466] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48ee44b093f4326a21d91a46610b10c in term 2.
I20250815 01:53:08.253652 1850 raft_consensus.cc:695] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [term 2 LEADER]: Becoming Leader. State: Replica: b48ee44b093f4326a21d91a46610b10c, State: Running, Role: LEADER
I20250815 01:53:08.254679 1850 consensus_queue.cc:237] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } }
I20250815 01:53:08.261947 1326 catalog_manager.cc:5582] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c reported cstate change: term changed from 1 to 2, leader changed from 366a46a9ba144f6c8e75e58fdd32e83e (127.0.106.193) to b48ee44b093f4326a21d91a46610b10c (127.0.106.194). New cstate: current_term: 2 leader_uuid: "b48ee44b093f4326a21d91a46610b10c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b48ee44b093f4326a21d91a46610b10c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 } health_report { overall_health: UNKNOWN } } }
I20250815 01:53:08.838632 1471 raft_consensus.cc:1273] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 2 FOLLOWER]: Refusing update from remote peer b48ee44b093f4326a21d91a46610b10c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 01:53:08.840111 1858 consensus_queue.cc:1035] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [LEADER]: Connected to new peer: Peer: permanent_uuid: "366a46a9ba144f6c8e75e58fdd32e83e" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250815 01:53:08.850785 1738 raft_consensus.cc:1273] T 1f45cdbbc91a4482b4740b024c0ecbde P 6f096cccbfe44c819b2752874360b680 [term 2 FOLLOWER]: Refusing update from remote peer b48ee44b093f4326a21d91a46610b10c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 01:53:08.853854 1863 consensus_queue.cc:1035] T 1f45cdbbc91a4482b4740b024c0ecbde P b48ee44b093f4326a21d91a46610b10c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f096cccbfe44c819b2752874360b680" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250815 01:53:10.750523 1471 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1f45cdbbc91a4482b4740b024c0ecbde"
dest_uuid: "366a46a9ba144f6c8e75e58fdd32e83e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43250
I20250815 01:53:10.751219 1471 raft_consensus.cc:604] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 2 FOLLOWER]: Received request to transfer leadership
I20250815 01:53:10.751570 1471 raft_consensus.cc:612] T 1f45cdbbc91a4482b4740b024c0ecbde P 366a46a9ba144f6c8e75e58fdd32e83e [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250815 01:53:11.788223 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 1387
I20250815 01:53:11.814251 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 1520
I20250815 01:53:11.838567 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 1653
I20250815 01:53:11.866602 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 1295
2025-08-15T01:53:11Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (16528 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250815 01:53:11.919925 427 test_util.cc:276] Using random seed: -1492936422
I20250815 01:53:11.923899 427 ts_itest-base.cc:115] Starting cluster with:
I20250815 01:53:11.924067 427 ts_itest-base.cc:116] --------------
I20250815 01:53:11.924234 427 ts_itest-base.cc:117] 3 tablet servers
I20250815 01:53:11.924360 427 ts_itest-base.cc:118] 3 replicas per TS
I20250815 01:53:11.924492 427 ts_itest-base.cc:119] --------------
2025-08-15T01:53:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T01:53:11Z Disabled control of system clock
I20250815 01:53:11.959992 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:38805
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:38805 with env {}
W20250815 01:53:12.259904 1898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:12.260551 1898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:12.261039 1898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:12.292771 1898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:53:12.293170 1898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:12.293444 1898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:53:12.293684 1898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:53:12.329872 1898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:38805
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:38805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:12.331229 1898 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:12.332808 1898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:12.343657 1904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:13.747428 1903 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 1898
W20250815 01:53:14.089350 1903 kernel_stack_watchdog.cc:198] Thread 1898 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:53:14.089876 1898 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.745s user 0.594s sys 1.145s
W20250815 01:53:12.344623 1905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:14.090416 1898 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.746s user 0.595s sys 1.145s
W20250815 01:53:14.090663 1906 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1746 milliseconds
W20250815 01:53:14.092139 1908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:14.092087 1898 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:14.095491 1898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:14.097992 1898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:14.099380 1898 hybrid_clock.cc:648] HybridClock initialized: now 1755222794099342 us; error 54 us; skew 500 ppm
I20250815 01:53:14.100170 1898 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:14.106240 1898 webserver.cc:480] Webserver started at http://127.0.106.254:35957/ using document root <none> and password file <none>
I20250815 01:53:14.107232 1898 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:14.107456 1898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:14.107863 1898 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:14.112313 1898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2"
format_stamp: "Formatted at 2025-08-15 01:53:14 on dist-test-slave-h491"
I20250815 01:53:14.113418 1898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2"
format_stamp: "Formatted at 2025-08-15 01:53:14 on dist-test-slave-h491"
I20250815 01:53:14.120777 1898 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250815 01:53:14.126813 1914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:14.127943 1898 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 01:53:14.128278 1898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2"
format_stamp: "Formatted at 2025-08-15 01:53:14 on dist-test-slave-h491"
I20250815 01:53:14.128625 1898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:14.181609 1898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:14.183149 1898 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:14.183607 1898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:14.254045 1898 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:38805
I20250815 01:53:14.254117 1965 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:38805 every 8 connection(s)
I20250815 01:53:14.256865 1898 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:53:14.262480 1966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:14.265261 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 1898
I20250815 01:53:14.265909 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 01:53:14.285552 1966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2: Bootstrap starting.
I20250815 01:53:14.292109 1966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:14.293812 1966 log.cc:826] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:14.298487 1966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2: No bootstrap required, opened a new log
I20250815 01:53:14.315915 1966 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38805 } }
I20250815 01:53:14.316799 1966 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:14.317039 1966 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4764dc08903f44d2b90c0c4fd6ad1fa2, State: Initialized, Role: FOLLOWER
I20250815 01:53:14.317745 1966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38805 } }
I20250815 01:53:14.318243 1966 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:53:14.318522 1966 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:53:14.318833 1966 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:14.322974 1966 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38805 } }
I20250815 01:53:14.323654 1966 leader_election.cc:304] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4764dc08903f44d2b90c0c4fd6ad1fa2; no voters:
I20250815 01:53:14.325438 1966 leader_election.cc:290] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:53:14.326195 1971 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:14.328568 1971 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 4764dc08903f44d2b90c0c4fd6ad1fa2, State: Running, Role: LEADER
I20250815 01:53:14.329406 1966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:53:14.329326 1971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38805 } }
I20250815 01:53:14.339625 1973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4764dc08903f44d2b90c0c4fd6ad1fa2. Latest consensus state: current_term: 1 leader_uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38805 } } }
I20250815 01:53:14.340076 1972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4764dc08903f44d2b90c0c4fd6ad1fa2" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38805 } } }
I20250815 01:53:14.340790 1973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [sys.catalog]: This master's current role is: LEADER
I20250815 01:53:14.340902 1972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2 [sys.catalog]: This master's current role is: LEADER
I20250815 01:53:14.346992 1980 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:53:14.361780 1980 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:53:14.379349 1980 catalog_manager.cc:1349] Generated new cluster ID: 2192c9f9deae4c7bb363cade6b9e9249
I20250815 01:53:14.379778 1980 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:53:14.416064 1980 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:53:14.417557 1980 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:53:14.431813 1980 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4764dc08903f44d2b90c0c4fd6ad1fa2: Generated new TSK 0
I20250815 01:53:14.432773 1980 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 01:53:14.442694 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38805
--builtin_ntp_servers=127.0.106.212:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 01:53:14.762265 1990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:14.762897 1990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:14.763407 1990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:14.795713 1990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:14.796595 1990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:53:14.832598 1990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:14.833920 1990 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:14.835605 1990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:14.848655 1997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:16.252013 1996 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 1990
W20250815 01:53:16.672190 1990 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.822s user 0.604s sys 1.036s
W20250815 01:53:16.672596 1990 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.823s user 0.604s sys 1.036s
W20250815 01:53:14.849886 1998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:16.673725 1999 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1823 milliseconds
I20250815 01:53:16.674906 1990 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 01:53:16.674925 2000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:16.678552 1990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:16.681144 1990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:16.682598 1990 hybrid_clock.cc:648] HybridClock initialized: now 1755222796682546 us; error 49 us; skew 500 ppm
I20250815 01:53:16.683635 1990 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:16.691087 1990 webserver.cc:480] Webserver started at http://127.0.106.193:42307/ using document root <none> and password file <none>
I20250815 01:53:16.692324 1990 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:16.692581 1990 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:16.693176 1990 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:16.699633 1990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1696f3db10a74395a9dc2c0871e1dc8c"
format_stamp: "Formatted at 2025-08-15 01:53:16 on dist-test-slave-h491"
I20250815 01:53:16.701195 1990 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1696f3db10a74395a9dc2c0871e1dc8c"
format_stamp: "Formatted at 2025-08-15 01:53:16 on dist-test-slave-h491"
I20250815 01:53:16.710511 1990 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20250815 01:53:16.717861 2007 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:16.719069 1990 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 01:53:16.719457 1990 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1696f3db10a74395a9dc2c0871e1dc8c"
format_stamp: "Formatted at 2025-08-15 01:53:16 on dist-test-slave-h491"
I20250815 01:53:16.719902 1990 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:16.786043 1990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:16.788058 1990 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:16.788594 1990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:16.791844 1990 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:16.797431 1990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:16.797717 1990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:16.798023 1990 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:16.798235 1990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:16.950397 1990 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:34635
I20250815 01:53:16.950547 2119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:34635 every 8 connection(s)
I20250815 01:53:16.952932 1990 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:53:16.962726 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 1990
I20250815 01:53:16.963379 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 01:53:16.970571 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38805
--builtin_ntp_servers=127.0.106.212:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:53:16.977412 2120 heartbeater.cc:344] Connected to a master server at 127.0.106.254:38805
I20250815 01:53:16.977878 2120 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:16.979097 2120 heartbeater.cc:507] Master 127.0.106.254:38805 requested a full tablet report, sending...
I20250815 01:53:16.982180 1931 ts_manager.cc:194] Registered new tserver with Master: 1696f3db10a74395a9dc2c0871e1dc8c (127.0.106.193:34635)
I20250815 01:53:16.985257 1931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:54691
W20250815 01:53:17.292510 2124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:17.292994 2124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:17.293439 2124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:17.325068 2124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:17.325867 2124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:53:17.360306 2124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:17.361528 2124 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:17.363174 2124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:17.376065 2130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:17.989917 2120 heartbeater.cc:499] Master 127.0.106.254:38805 was elected leader, sending a full tablet report...
W20250815 01:53:18.780361 2129 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 2124
W20250815 01:53:19.147701 2124 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.771s user 0.655s sys 1.115s
W20250815 01:53:19.150027 2124 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.774s user 0.655s sys 1.115s
W20250815 01:53:17.376571 2131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:19.150425 2133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:19.150776 2132 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1773 milliseconds
I20250815 01:53:19.150696 2124 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:19.155575 2124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:19.157606 2124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:19.158948 2124 hybrid_clock.cc:648] HybridClock initialized: now 1755222799158931 us; error 74 us; skew 500 ppm
I20250815 01:53:19.159739 2124 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:19.165917 2124 webserver.cc:480] Webserver started at http://127.0.106.194:43049/ using document root <none> and password file <none>
I20250815 01:53:19.166863 2124 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:19.167073 2124 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:19.167521 2124 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:19.171855 2124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ad4fb2a630ab49988d2569565f913133"
format_stamp: "Formatted at 2025-08-15 01:53:19 on dist-test-slave-h491"
I20250815 01:53:19.172951 2124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ad4fb2a630ab49988d2569565f913133"
format_stamp: "Formatted at 2025-08-15 01:53:19 on dist-test-slave-h491"
I20250815 01:53:19.179949 2124 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20250815 01:53:19.185650 2140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:19.186915 2124 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250815 01:53:19.187268 2124 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ad4fb2a630ab49988d2569565f913133"
format_stamp: "Formatted at 2025-08-15 01:53:19 on dist-test-slave-h491"
I20250815 01:53:19.187611 2124 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:19.237079 2124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:19.238621 2124 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:19.239048 2124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:19.241489 2124 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:19.245568 2124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:19.245783 2124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:19.246037 2124 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:19.246253 2124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250815 01:53:19.384505 2124 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:38893
I20250815 01:53:19.384610 2252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:38893 every 8 connection(s)
I20250815 01:53:19.387079 2124 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 01:53:19.397775 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 2124
I20250815 01:53:19.398295 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 01:53:19.404547 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38805
--builtin_ntp_servers=127.0.106.212:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:53:19.415362 2253 heartbeater.cc:344] Connected to a master server at 127.0.106.254:38805
I20250815 01:53:19.415799 2253 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:19.416831 2253 heartbeater.cc:507] Master 127.0.106.254:38805 requested a full tablet report, sending...
I20250815 01:53:19.419191 1931 ts_manager.cc:194] Registered new tserver with Master: ad4fb2a630ab49988d2569565f913133 (127.0.106.194:38893)
I20250815 01:53:19.421185 1931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:58171
W20250815 01:53:19.702109 2257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:19.702659 2257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:19.703111 2257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:19.734977 2257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:19.735793 2257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:53:19.770948 2257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38805
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:19.772207 2257 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:19.773738 2257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:19.785106 2263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:20.424579 2253 heartbeater.cc:499] Master 127.0.106.254:38805 was elected leader, sending a full tablet report...
W20250815 01:53:19.786696 2264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:21.020766 2265 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1232 milliseconds
W20250815 01:53:21.022557 2266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:21.024888 2257 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.239s user 0.002s sys 0.003s
W20250815 01:53:21.025369 2257 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.240s user 0.004s sys 0.004s
I20250815 01:53:21.025710 2257 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:21.027328 2257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:21.029693 2257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:21.031087 2257 hybrid_clock.cc:648] HybridClock initialized: now 1755222801031040 us; error 62 us; skew 500 ppm
I20250815 01:53:21.031903 2257 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:21.038928 2257 webserver.cc:480] Webserver started at http://127.0.106.195:35969/ using document root <none> and password file <none>
I20250815 01:53:21.039979 2257 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:21.040225 2257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:21.040750 2257 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:21.045567 2257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0b71afb6990f4f159f3e3780dd289c31"
format_stamp: "Formatted at 2025-08-15 01:53:21 on dist-test-slave-h491"
I20250815 01:53:21.046721 2257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0b71afb6990f4f159f3e3780dd289c31"
format_stamp: "Formatted at 2025-08-15 01:53:21 on dist-test-slave-h491"
I20250815 01:53:21.054420 2257 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250815 01:53:21.060510 2273 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:21.061803 2257 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 01:53:21.062136 2257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0b71afb6990f4f159f3e3780dd289c31"
format_stamp: "Formatted at 2025-08-15 01:53:21 on dist-test-slave-h491"
I20250815 01:53:21.062525 2257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:21.141345 2257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:21.142889 2257 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:21.143324 2257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:21.145880 2257 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:21.149966 2257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:21.150174 2257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:21.150470 2257 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:21.150624 2257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:21.286233 2257 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:39189
I20250815 01:53:21.286404 2385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:39189 every 8 connection(s)
I20250815 01:53:21.288740 2257 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 01:53:21.296684 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 2257
I20250815 01:53:21.297071 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 01:53:21.309681 2386 heartbeater.cc:344] Connected to a master server at 127.0.106.254:38805
I20250815 01:53:21.310120 2386 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:21.311137 2386 heartbeater.cc:507] Master 127.0.106.254:38805 requested a full tablet report, sending...
I20250815 01:53:21.313264 1931 ts_manager.cc:194] Registered new tserver with Master: 0b71afb6990f4f159f3e3780dd289c31 (127.0.106.195:39189)
I20250815 01:53:21.314715 1931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:59069
I20250815 01:53:21.316628 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:53:21.350131 1931 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51208:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 01:53:21.368836 1931 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 01:53:21.418282 2055 tablet_service.cc:1468] Processing CreateTablet for tablet e72a83ecb026490dbcad04149620bc85 (DEFAULT_TABLE table=TestTable [id=3623ffa03df34870aceae818521ffa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:21.420130 2055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72a83ecb026490dbcad04149620bc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:21.420886 2321 tablet_service.cc:1468] Processing CreateTablet for tablet e72a83ecb026490dbcad04149620bc85 (DEFAULT_TABLE table=TestTable [id=3623ffa03df34870aceae818521ffa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:21.423009 2321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72a83ecb026490dbcad04149620bc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:21.430590 2188 tablet_service.cc:1468] Processing CreateTablet for tablet e72a83ecb026490dbcad04149620bc85 (DEFAULT_TABLE table=TestTable [id=3623ffa03df34870aceae818521ffa74]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:21.432744 2188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e72a83ecb026490dbcad04149620bc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:21.447851 2405 tablet_bootstrap.cc:492] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c: Bootstrap starting.
I20250815 01:53:21.452461 2406 tablet_bootstrap.cc:492] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31: Bootstrap starting.
I20250815 01:53:21.456317 2405 tablet_bootstrap.cc:654] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:21.458056 2407 tablet_bootstrap.cc:492] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133: Bootstrap starting.
I20250815 01:53:21.458676 2405 log.cc:826] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:21.460027 2406 tablet_bootstrap.cc:654] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:21.462579 2406 log.cc:826] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:21.463796 2405 tablet_bootstrap.cc:492] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c: No bootstrap required, opened a new log
I20250815 01:53:21.464247 2405 ts_tablet_manager.cc:1397] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20250815 01:53:21.465739 2407 tablet_bootstrap.cc:654] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:21.468102 2407 log.cc:826] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:21.469133 2406 tablet_bootstrap.cc:492] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31: No bootstrap required, opened a new log
I20250815 01:53:21.469710 2406 ts_tablet_manager.cc:1397] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20250815 01:53:21.473883 2407 tablet_bootstrap.cc:492] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133: No bootstrap required, opened a new log
I20250815 01:53:21.474467 2407 ts_tablet_manager.cc:1397] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20250815 01:53:21.485239 2405 raft_consensus.cc:357] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.485946 2405 raft_consensus.cc:383] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:21.486222 2405 raft_consensus.cc:738] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1696f3db10a74395a9dc2c0871e1dc8c, State: Initialized, Role: FOLLOWER
I20250815 01:53:21.487025 2405 consensus_queue.cc:260] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.495741 2405 ts_tablet_manager.cc:1428] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c: Time spent starting tablet: real 0.031s user 0.021s sys 0.004s
I20250815 01:53:21.496657 2406 raft_consensus.cc:357] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.497584 2406 raft_consensus.cc:383] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:21.497941 2406 raft_consensus.cc:738] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b71afb6990f4f159f3e3780dd289c31, State: Initialized, Role: FOLLOWER
I20250815 01:53:21.499027 2406 consensus_queue.cc:260] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.501195 2407 raft_consensus.cc:357] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.501916 2407 raft_consensus.cc:383] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:21.502142 2407 raft_consensus.cc:738] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4fb2a630ab49988d2569565f913133, State: Initialized, Role: FOLLOWER
I20250815 01:53:21.503414 2407 consensus_queue.cc:260] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.504230 2386 heartbeater.cc:499] Master 127.0.106.254:38805 was elected leader, sending a full tablet report...
I20250815 01:53:21.507786 2407 ts_tablet_manager.cc:1428] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133: Time spent starting tablet: real 0.033s user 0.031s sys 0.001s
I20250815 01:53:21.509534 2406 ts_tablet_manager.cc:1428] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31: Time spent starting tablet: real 0.039s user 0.023s sys 0.009s
W20250815 01:53:21.543251 2387 tablet.cc:2378] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:21.592025 2412 raft_consensus.cc:491] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:53:21.592554 2412 raft_consensus.cc:513] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.594928 2412 leader_election.cc:290] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1696f3db10a74395a9dc2c0871e1dc8c (127.0.106.193:34635), ad4fb2a630ab49988d2569565f913133 (127.0.106.194:38893)
I20250815 01:53:21.603027 2075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e72a83ecb026490dbcad04149620bc85" candidate_uuid: "0b71afb6990f4f159f3e3780dd289c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" is_pre_election: true
I20250815 01:53:21.603890 2075 raft_consensus.cc:2466] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b71afb6990f4f159f3e3780dd289c31 in term 0.
I20250815 01:53:21.605167 2208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e72a83ecb026490dbcad04149620bc85" candidate_uuid: "0b71afb6990f4f159f3e3780dd289c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4fb2a630ab49988d2569565f913133" is_pre_election: true
I20250815 01:53:21.605304 2276 leader_election.cc:304] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b71afb6990f4f159f3e3780dd289c31, 1696f3db10a74395a9dc2c0871e1dc8c; no voters:
I20250815 01:53:21.606098 2208 raft_consensus.cc:2466] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b71afb6990f4f159f3e3780dd289c31 in term 0.
I20250815 01:53:21.606190 2412 raft_consensus.cc:2802] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 01:53:21.606554 2412 raft_consensus.cc:491] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:53:21.606814 2412 raft_consensus.cc:3058] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:21.611327 2412 raft_consensus.cc:513] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.612785 2412 leader_election.cc:290] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [CANDIDATE]: Term 1 election: Requested vote from peers 1696f3db10a74395a9dc2c0871e1dc8c (127.0.106.193:34635), ad4fb2a630ab49988d2569565f913133 (127.0.106.194:38893)
I20250815 01:53:21.613662 2075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e72a83ecb026490dbcad04149620bc85" candidate_uuid: "0b71afb6990f4f159f3e3780dd289c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1696f3db10a74395a9dc2c0871e1dc8c"
I20250815 01:53:21.613842 2208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e72a83ecb026490dbcad04149620bc85" candidate_uuid: "0b71afb6990f4f159f3e3780dd289c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4fb2a630ab49988d2569565f913133"
I20250815 01:53:21.614223 2075 raft_consensus.cc:3058] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:21.614341 2208 raft_consensus.cc:3058] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:21.621359 2075 raft_consensus.cc:2466] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b71afb6990f4f159f3e3780dd289c31 in term 1.
I20250815 01:53:21.621357 2208 raft_consensus.cc:2466] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b71afb6990f4f159f3e3780dd289c31 in term 1.
I20250815 01:53:21.622653 2274 leader_election.cc:304] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b71afb6990f4f159f3e3780dd289c31, ad4fb2a630ab49988d2569565f913133; no voters:
I20250815 01:53:21.623383 2412 raft_consensus.cc:2802] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:21.624897 2412 raft_consensus.cc:695] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [term 1 LEADER]: Becoming Leader. State: Replica: 0b71afb6990f4f159f3e3780dd289c31, State: Running, Role: LEADER
I20250815 01:53:21.625653 2412 consensus_queue.cc:237] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } }
I20250815 01:53:21.635664 1930 catalog_manager.cc:5582] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b71afb6990f4f159f3e3780dd289c31 (127.0.106.195). New cstate: current_term: 1 leader_uuid: "0b71afb6990f4f159f3e3780dd289c31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } health_report { overall_health: UNKNOWN } } }
W20250815 01:53:21.643905 2254 tablet.cc:2378] T e72a83ecb026490dbcad04149620bc85 P ad4fb2a630ab49988d2569565f913133: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:21.670672 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:53:21.674041 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1696f3db10a74395a9dc2c0871e1dc8c to finish bootstrapping
I20250815 01:53:21.686702 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ad4fb2a630ab49988d2569565f913133 to finish bootstrapping
I20250815 01:53:21.697403 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0b71afb6990f4f159f3e3780dd289c31 to finish bootstrapping
I20250815 01:53:21.711309 1930 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51208:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250815 01:53:21.712853 1930 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250815 01:53:21.713212 2121 tablet.cc:2378] T e72a83ecb026490dbcad04149620bc85 P 1696f3db10a74395a9dc2c0871e1dc8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:21.729810 2055 tablet_service.cc:1468] Processing CreateTablet for tablet 0d109d71502745748477466bc0f37095 (DEFAULT_TABLE table=TestAnotherTable [id=b350bf4b31b04a2abbda8c8b31a85f9a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 01:53:21.730178 2188 tablet_service.cc:1468] Processing CreateTablet for tablet 0d109d71502745748477466bc0f37095 (DEFAULT_TABLE table=TestAnotherTable [id=b350bf4b31b04a2abbda8c8b31a85f9a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 01:53:21.730958 2055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d109d71502745748477466bc0f37095. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:21.731319 2188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d109d71502745748477466bc0f37095. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:21.737736 2321 tablet_service.cc:1468] Processing CreateTablet for tablet 0d109d71502745748477466bc0f37095 (DEFAULT_TABLE table=TestAnotherTable [id=b350bf4b31b04a2abbda8c8b31a85f9a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 01:53:21.739097 2321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d109d71502745748477466bc0f37095. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:21.741879 2407 tablet_bootstrap.cc:492] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133: Bootstrap starting.
I20250815 01:53:21.742790 2405 tablet_bootstrap.cc:492] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c: Bootstrap starting.
I20250815 01:53:21.747392 2407 tablet_bootstrap.cc:654] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:21.748725 2405 tablet_bootstrap.cc:654] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:21.753835 2406 tablet_bootstrap.cc:492] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31: Bootstrap starting.
I20250815 01:53:21.754652 2407 tablet_bootstrap.cc:492] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133: No bootstrap required, opened a new log
I20250815 01:53:21.754998 2407 ts_tablet_manager.cc:1397] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250815 01:53:21.755084 2405 tablet_bootstrap.cc:492] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c: No bootstrap required, opened a new log
I20250815 01:53:21.755501 2405 ts_tablet_manager.cc:1397] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250815 01:53:21.756932 2407 raft_consensus.cc:357] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.757431 2407 raft_consensus.cc:383] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:21.757738 2407 raft_consensus.cc:738] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4fb2a630ab49988d2569565f913133, State: Initialized, Role: FOLLOWER
I20250815 01:53:21.758204 2405 raft_consensus.cc:357] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.758980 2405 raft_consensus.cc:383] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:21.758627 2407 consensus_queue.cc:260] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.759305 2405 raft_consensus.cc:738] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1696f3db10a74395a9dc2c0871e1dc8c, State: Initialized, Role: FOLLOWER
I20250815 01:53:21.759917 2406 tablet_bootstrap.cc:654] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:21.760097 2405 consensus_queue.cc:260] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.761605 2407 ts_tablet_manager.cc:1428] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250815 01:53:21.762671 2405 ts_tablet_manager.cc:1428] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250815 01:53:21.767211 2406 tablet_bootstrap.cc:492] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31: No bootstrap required, opened a new log
I20250815 01:53:21.767647 2406 ts_tablet_manager.cc:1397] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20250815 01:53:21.770066 2406 raft_consensus.cc:357] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.770608 2406 raft_consensus.cc:383] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:21.770834 2406 raft_consensus.cc:738] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b71afb6990f4f159f3e3780dd289c31, State: Initialized, Role: FOLLOWER
I20250815 01:53:21.771368 2406 consensus_queue.cc:260] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.773382 2406 ts_tablet_manager.cc:1428] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250815 01:53:21.800956 2413 raft_consensus.cc:491] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:53:21.801405 2413 raft_consensus.cc:513] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.804195 2413 leader_election.cc:290] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1696f3db10a74395a9dc2c0871e1dc8c (127.0.106.193:34635), 0b71afb6990f4f159f3e3780dd289c31 (127.0.106.195:39189)
I20250815 01:53:21.816190 2075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d109d71502745748477466bc0f37095" candidate_uuid: "ad4fb2a630ab49988d2569565f913133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" is_pre_election: true
I20250815 01:53:21.816695 2075 raft_consensus.cc:2466] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad4fb2a630ab49988d2569565f913133 in term 0.
I20250815 01:53:21.816567 2341 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d109d71502745748477466bc0f37095" candidate_uuid: "ad4fb2a630ab49988d2569565f913133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b71afb6990f4f159f3e3780dd289c31" is_pre_election: true
I20250815 01:53:21.817193 2341 raft_consensus.cc:2466] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad4fb2a630ab49988d2569565f913133 in term 0.
I20250815 01:53:21.817735 2143 leader_election.cc:304] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1696f3db10a74395a9dc2c0871e1dc8c, ad4fb2a630ab49988d2569565f913133; no voters:
I20250815 01:53:21.818619 2413 raft_consensus.cc:2802] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 01:53:21.818949 2413 raft_consensus.cc:491] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:53:21.819175 2413 raft_consensus.cc:3058] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:21.823583 2413 raft_consensus.cc:513] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.825129 2413 leader_election.cc:290] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [CANDIDATE]: Term 1 election: Requested vote from peers 1696f3db10a74395a9dc2c0871e1dc8c (127.0.106.193:34635), 0b71afb6990f4f159f3e3780dd289c31 (127.0.106.195:39189)
I20250815 01:53:21.825862 2075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d109d71502745748477466bc0f37095" candidate_uuid: "ad4fb2a630ab49988d2569565f913133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1696f3db10a74395a9dc2c0871e1dc8c"
I20250815 01:53:21.826162 2341 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d109d71502745748477466bc0f37095" candidate_uuid: "ad4fb2a630ab49988d2569565f913133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b71afb6990f4f159f3e3780dd289c31"
I20250815 01:53:21.826464 2075 raft_consensus.cc:3058] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:21.826736 2341 raft_consensus.cc:3058] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:21.833154 2075 raft_consensus.cc:2466] T 0d109d71502745748477466bc0f37095 P 1696f3db10a74395a9dc2c0871e1dc8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad4fb2a630ab49988d2569565f913133 in term 1.
I20250815 01:53:21.833693 2341 raft_consensus.cc:2466] T 0d109d71502745748477466bc0f37095 P 0b71afb6990f4f159f3e3780dd289c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad4fb2a630ab49988d2569565f913133 in term 1.
I20250815 01:53:21.834293 2143 leader_election.cc:304] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1696f3db10a74395a9dc2c0871e1dc8c, ad4fb2a630ab49988d2569565f913133; no voters:
I20250815 01:53:21.835109 2413 raft_consensus.cc:2802] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:21.836678 2413 raft_consensus.cc:695] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [term 1 LEADER]: Becoming Leader. State: Replica: ad4fb2a630ab49988d2569565f913133, State: Running, Role: LEADER
I20250815 01:53:21.837469 2413 consensus_queue.cc:237] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } }
I20250815 01:53:21.848835 1930 catalog_manager.cc:5582] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad4fb2a630ab49988d2569565f913133 (127.0.106.194). New cstate: current_term: 1 leader_uuid: "ad4fb2a630ab49988d2569565f913133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 } health_report { overall_health: UNKNOWN } } }
I20250815 01:53:22.068666 2412 consensus_queue.cc:1035] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:53:22.095134 2415 consensus_queue.cc:1035] T e72a83ecb026490dbcad04149620bc85 P 0b71afb6990f4f159f3e3780dd289c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad4fb2a630ab49988d2569565f913133" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 01:53:22.193902 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:22.194535 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:22.226094 2425 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250815 01:53:22.257567 2413 consensus_queue.cc:1035] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1696f3db10a74395a9dc2c0871e1dc8c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 34635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:53:22.272541 2423 consensus_queue.cc:1035] T 0d109d71502745748477466bc0f37095 P ad4fb2a630ab49988d2569565f913133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b71afb6990f4f159f3e3780dd289c31" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 01:53:23.643776 2425 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.334s user 0.512s sys 0.810s
W20250815 01:53:23.644078 2425 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.334s user 0.512s sys 0.810s
W20250815 01:53:25.034240 2452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:25.034871 2452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:25.067883 2452 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 01:53:26.372738 2452 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.265s user 0.466s sys 0.797s
W20250815 01:53:26.373057 2452 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.266s user 0.466s sys 0.797s
W20250815 01:53:27.769129 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:27.769703 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:27.801183 2471 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 01:53:29.087813 2471 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.248s user 0.418s sys 0.828s
W20250815 01:53:29.088222 2471 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.248s user 0.418s sys 0.828s
W20250815 01:53:30.473101 2486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:30.473649 2486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:30.504880 2486 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 01:53:31.784933 2486 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.241s user 0.529s sys 0.710s
W20250815 01:53:31.785239 2486 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.242s user 0.529s sys 0.710s
I20250815 01:53:32.861128 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 1990
I20250815 01:53:32.886382 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 2124
I20250815 01:53:32.912678 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 2257
I20250815 01:53:32.940065 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 1898
2025-08-15T01:53:32Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (21082 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250815 01:53:33.002005 427 test_util.cc:276] Using random seed: -1471854342
I20250815 01:53:33.006560 427 ts_itest-base.cc:115] Starting cluster with:
I20250815 01:53:33.006758 427 ts_itest-base.cc:116] --------------
I20250815 01:53:33.006888 427 ts_itest-base.cc:117] 3 tablet servers
I20250815 01:53:33.007006 427 ts_itest-base.cc:118] 3 replicas per TS
I20250815 01:53:33.007114 427 ts_itest-base.cc:119] --------------
2025-08-15T01:53:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T01:53:33Z Disabled control of system clock
I20250815 01:53:33.047036 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:34727
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:34727
--superuser_acl=no-such-user with env {}
W20250815 01:53:33.356634 2508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:33.357247 2508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:33.357688 2508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:33.388626 2508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:53:33.388928 2508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:33.389134 2508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:53:33.389325 2508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:53:33.424350 2508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:34727
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:33.425568 2508 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:33.427187 2508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:33.437669 2514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:33.439527 2515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:34.951049 2516 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1510 milliseconds
W20250815 01:53:34.841393 2513 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ec0 after lost signal to thread 2508
W20250815 01:53:34.950210 2508 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.512s user 0.532s sys 0.973s
W20250815 01:53:34.951927 2508 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.514s user 0.533s sys 0.974s
W20250815 01:53:34.953025 2517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:34.952997 2508 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:34.959221 2508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:34.962406 2508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:34.963847 2508 hybrid_clock.cc:648] HybridClock initialized: now 1755222814963809 us; error 49 us; skew 500 ppm
I20250815 01:53:34.964952 2508 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:34.972720 2508 webserver.cc:480] Webserver started at http://127.0.106.254:42613/ using document root <none> and password file <none>
I20250815 01:53:34.974114 2508 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:34.974442 2508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:34.975064 2508 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:34.980293 2508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "95a163c42b3a46efa450cc06d047ab2a"
format_stamp: "Formatted at 2025-08-15 01:53:34 on dist-test-slave-h491"
I20250815 01:53:34.981400 2508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "95a163c42b3a46efa450cc06d047ab2a"
format_stamp: "Formatted at 2025-08-15 01:53:34 on dist-test-slave-h491"
I20250815 01:53:34.988453 2508 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20250815 01:53:34.993927 2525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:34.995144 2508 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250815 01:53:34.995493 2508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "95a163c42b3a46efa450cc06d047ab2a"
format_stamp: "Formatted at 2025-08-15 01:53:34 on dist-test-slave-h491"
I20250815 01:53:34.995857 2508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:35.045289 2508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:35.046828 2508 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:35.047276 2508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:35.115262 2508 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:34727
I20250815 01:53:35.115350 2576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:34727 every 8 connection(s)
I20250815 01:53:35.117995 2508 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:53:35.123203 2577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:35.124562 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 2508
I20250815 01:53:35.125293 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 01:53:35.145008 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a: Bootstrap starting.
I20250815 01:53:35.151221 2577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:35.153137 2577 log.cc:826] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:35.157306 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a: No bootstrap required, opened a new log
I20250815 01:53:35.175061 2577 raft_consensus.cc:357] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a163c42b3a46efa450cc06d047ab2a" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 34727 } }
I20250815 01:53:35.175767 2577 raft_consensus.cc:383] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:35.176007 2577 raft_consensus.cc:738] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a163c42b3a46efa450cc06d047ab2a, State: Initialized, Role: FOLLOWER
I20250815 01:53:35.176723 2577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a163c42b3a46efa450cc06d047ab2a" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 34727 } }
I20250815 01:53:35.177235 2577 raft_consensus.cc:397] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:53:35.177500 2577 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:53:35.177806 2577 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:35.182230 2577 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a163c42b3a46efa450cc06d047ab2a" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 34727 } }
I20250815 01:53:35.183318 2577 leader_election.cc:304] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95a163c42b3a46efa450cc06d047ab2a; no voters:
I20250815 01:53:35.184995 2577 leader_election.cc:290] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:53:35.185673 2582 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:35.187844 2582 raft_consensus.cc:695] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [term 1 LEADER]: Becoming Leader. State: Replica: 95a163c42b3a46efa450cc06d047ab2a, State: Running, Role: LEADER
I20250815 01:53:35.188601 2582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a163c42b3a46efa450cc06d047ab2a" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 34727 } }
I20250815 01:53:35.189738 2577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:53:35.198997 2584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95a163c42b3a46efa450cc06d047ab2a. Latest consensus state: current_term: 1 leader_uuid: "95a163c42b3a46efa450cc06d047ab2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a163c42b3a46efa450cc06d047ab2a" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 34727 } } }
I20250815 01:53:35.199730 2584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [sys.catalog]: This master's current role is: LEADER
I20250815 01:53:35.199467 2583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95a163c42b3a46efa450cc06d047ab2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a163c42b3a46efa450cc06d047ab2a" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 34727 } } }
I20250815 01:53:35.200016 2583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a [sys.catalog]: This master's current role is: LEADER
I20250815 01:53:35.203737 2590 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:53:35.212218 2590 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:53:35.229091 2590 catalog_manager.cc:1349] Generated new cluster ID: 585dcc8b71924682b508d73f9f8e81e2
I20250815 01:53:35.229399 2590 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:53:35.246786 2590 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:53:35.248262 2590 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:53:35.260331 2590 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 95a163c42b3a46efa450cc06d047ab2a: Generated new TSK 0
I20250815 01:53:35.261441 2590 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 01:53:35.276916 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:34727
--builtin_ntp_servers=127.0.106.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 01:53:35.579181 2601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:35.579703 2601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:35.580212 2601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:35.611827 2601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:35.612684 2601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:53:35.648339 2601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:34727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:35.649590 2601 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:35.651187 2601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:35.664134 2607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:37.064817 2606 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 2601
W20250815 01:53:35.665192 2608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:37.474614 2606 kernel_stack_watchdog.cc:198] Thread 2601 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:53:37.475318 2601 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.809s user 0.592s sys 1.127s
W20250815 01:53:37.475767 2601 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.810s user 0.592s sys 1.127s
W20250815 01:53:37.477808 2610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:37.483386 2609 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1815 milliseconds
I20250815 01:53:37.483479 2601 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:37.484879 2601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:37.487442 2601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:37.488948 2601 hybrid_clock.cc:648] HybridClock initialized: now 1755222817488906 us; error 40 us; skew 500 ppm
I20250815 01:53:37.490028 2601 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:37.497694 2601 webserver.cc:480] Webserver started at http://127.0.106.193:37853/ using document root <none> and password file <none>
I20250815 01:53:37.498981 2601 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:37.499264 2601 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:37.499835 2601 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:37.506471 2601 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fa88fb9f464e429abe650a49dca4f072"
format_stamp: "Formatted at 2025-08-15 01:53:37 on dist-test-slave-h491"
I20250815 01:53:37.508078 2601 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fa88fb9f464e429abe650a49dca4f072"
format_stamp: "Formatted at 2025-08-15 01:53:37 on dist-test-slave-h491"
I20250815 01:53:37.517489 2601 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20250815 01:53:37.525132 2617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:37.526332 2601 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.006s
I20250815 01:53:37.526777 2601 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fa88fb9f464e429abe650a49dca4f072"
format_stamp: "Formatted at 2025-08-15 01:53:37 on dist-test-slave-h491"
I20250815 01:53:37.527240 2601 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:37.595896 2601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:37.597353 2601 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:37.597800 2601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:37.600386 2601 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:37.605190 2601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:37.605412 2601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:37.605672 2601 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:37.605834 2601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:37.767470 2601 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:45207
I20250815 01:53:37.767580 2729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:45207 every 8 connection(s)
I20250815 01:53:37.770102 2601 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:53:37.777274 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 2601
I20250815 01:53:37.777879 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 01:53:37.786789 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:34727
--builtin_ntp_servers=127.0.106.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:53:37.809146 2730 heartbeater.cc:344] Connected to a master server at 127.0.106.254:34727
I20250815 01:53:37.809749 2730 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:37.811687 2730 heartbeater.cc:507] Master 127.0.106.254:34727 requested a full tablet report, sending...
I20250815 01:53:37.814720 2542 ts_manager.cc:194] Registered new tserver with Master: fa88fb9f464e429abe650a49dca4f072 (127.0.106.193:45207)
I20250815 01:53:37.817433 2542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:46809
W20250815 01:53:38.090050 2734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:38.090571 2734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:38.091041 2734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:38.132923 2734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:38.134305 2734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:53:38.172267 2734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:34727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:38.173748 2734 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:38.175351 2734 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:38.187718 2740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:38.821377 2730 heartbeater.cc:499] Master 127.0.106.254:34727 was elected leader, sending a full tablet report...
W20250815 01:53:38.188483 2741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:39.411068 2742 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1220 milliseconds
W20250815 01:53:39.411405 2734 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.223s user 0.353s sys 0.862s
W20250815 01:53:39.411823 2734 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.224s user 0.354s sys 0.863s
W20250815 01:53:39.411780 2743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:39.412117 2734 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:39.413224 2734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:39.415423 2734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:39.416805 2734 hybrid_clock.cc:648] HybridClock initialized: now 1755222819416776 us; error 40 us; skew 500 ppm
I20250815 01:53:39.417593 2734 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:39.424674 2734 webserver.cc:480] Webserver started at http://127.0.106.194:34373/ using document root <none> and password file <none>
I20250815 01:53:39.425626 2734 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:39.425855 2734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:39.426316 2734 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:39.430984 2734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "224e0981647c4baba5a6cf2f6b7ca9b0"
format_stamp: "Formatted at 2025-08-15 01:53:39 on dist-test-slave-h491"
I20250815 01:53:39.432118 2734 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "224e0981647c4baba5a6cf2f6b7ca9b0"
format_stamp: "Formatted at 2025-08-15 01:53:39 on dist-test-slave-h491"
I20250815 01:53:39.439726 2734 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250815 01:53:39.446039 2750 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:39.447295 2734 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250815 01:53:39.447638 2734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "224e0981647c4baba5a6cf2f6b7ca9b0"
format_stamp: "Formatted at 2025-08-15 01:53:39 on dist-test-slave-h491"
I20250815 01:53:39.447983 2734 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:39.514568 2734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:39.516041 2734 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:39.516475 2734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:39.519544 2734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:39.523898 2734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:39.524111 2734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:39.524361 2734 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:39.524528 2734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:39.657100 2734 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:41971
I20250815 01:53:39.657217 2862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:41971 every 8 connection(s)
I20250815 01:53:39.659832 2734 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 01:53:39.663018 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 2734
I20250815 01:53:39.663542 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 01:53:39.670948 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:34727
--builtin_ntp_servers=127.0.106.212:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:53:39.684223 2863 heartbeater.cc:344] Connected to a master server at 127.0.106.254:34727
I20250815 01:53:39.684798 2863 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:39.686245 2863 heartbeater.cc:507] Master 127.0.106.254:34727 requested a full tablet report, sending...
I20250815 01:53:39.688854 2542 ts_manager.cc:194] Registered new tserver with Master: 224e0981647c4baba5a6cf2f6b7ca9b0 (127.0.106.194:41971)
I20250815 01:53:39.690129 2542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:45025
W20250815 01:53:39.984207 2867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:39.984722 2867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:39.985299 2867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:40.017360 2867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:40.018430 2867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:53:40.054019 2867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:34727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:40.055357 2867 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:40.057003 2867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:40.069195 2873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:40.694233 2863 heartbeater.cc:499] Master 127.0.106.254:34727 was elected leader, sending a full tablet report...
W20250815 01:53:40.069558 2874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:41.299703 2876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:41.302210 2867 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.232s user 0.000s sys 0.007s
W20250815 01:53:41.302531 2867 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.232s user 0.000s sys 0.011s
I20250815 01:53:41.302752 2867 server_base.cc:1047] running on GCE node
I20250815 01:53:41.303809 2867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:41.306401 2867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:41.307864 2867 hybrid_clock.cc:648] HybridClock initialized: now 1755222821307810 us; error 76 us; skew 500 ppm
I20250815 01:53:41.308779 2867 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:41.316884 2867 webserver.cc:480] Webserver started at http://127.0.106.195:36725/ using document root <none> and password file <none>
I20250815 01:53:41.317795 2867 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:41.317998 2867 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:41.318485 2867 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:41.324790 2867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "52eba469cb73439eb8322778ab0c8177"
format_stamp: "Formatted at 2025-08-15 01:53:41 on dist-test-slave-h491"
I20250815 01:53:41.325934 2867 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "52eba469cb73439eb8322778ab0c8177"
format_stamp: "Formatted at 2025-08-15 01:53:41 on dist-test-slave-h491"
I20250815 01:53:41.333930 2867 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250815 01:53:41.340317 2883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:41.341410 2867 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 01:53:41.341728 2867 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "52eba469cb73439eb8322778ab0c8177"
format_stamp: "Formatted at 2025-08-15 01:53:41 on dist-test-slave-h491"
I20250815 01:53:41.342056 2867 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:41.415452 2867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:41.416941 2867 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:41.417361 2867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:41.419896 2867 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:41.424050 2867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:41.424247 2867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:41.424497 2867 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:41.424651 2867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:41.558318 2867 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:42821
I20250815 01:53:41.558437 2995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:42821 every 8 connection(s)
I20250815 01:53:41.560905 2867 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 01:53:41.570705 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 2867
I20250815 01:53:41.571116 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 01:53:41.581632 2996 heartbeater.cc:344] Connected to a master server at 127.0.106.254:34727
I20250815 01:53:41.582094 2996 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:41.583163 2996 heartbeater.cc:507] Master 127.0.106.254:34727 requested a full tablet report, sending...
I20250815 01:53:41.585266 2542 ts_manager.cc:194] Registered new tserver with Master: 52eba469cb73439eb8322778ab0c8177 (127.0.106.195:42821)
I20250815 01:53:41.586508 2542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:42595
I20250815 01:53:41.590855 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:53:41.624675 2542 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48300:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 01:53:41.643738 2542 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 01:53:41.699023 2665 tablet_service.cc:1468] Processing CreateTablet for tablet d7899958bdce4f579e3bef5ba87c6114 (DEFAULT_TABLE table=TestTable [id=79ef0c121a7847a1bf9f9be51ae51c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:41.702775 2665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7899958bdce4f579e3bef5ba87c6114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:41.704644 2798 tablet_service.cc:1468] Processing CreateTablet for tablet d7899958bdce4f579e3bef5ba87c6114 (DEFAULT_TABLE table=TestTable [id=79ef0c121a7847a1bf9f9be51ae51c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:41.706264 2798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7899958bdce4f579e3bef5ba87c6114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:41.705977 2931 tablet_service.cc:1468] Processing CreateTablet for tablet d7899958bdce4f579e3bef5ba87c6114 (DEFAULT_TABLE table=TestTable [id=79ef0c121a7847a1bf9f9be51ae51c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:41.707939 2931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7899958bdce4f579e3bef5ba87c6114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:41.727702 3015 tablet_bootstrap.cc:492] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0: Bootstrap starting.
I20250815 01:53:41.731192 3016 tablet_bootstrap.cc:492] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072: Bootstrap starting.
I20250815 01:53:41.737360 3015 tablet_bootstrap.cc:654] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:41.738288 3017 tablet_bootstrap.cc:492] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177: Bootstrap starting.
I20250815 01:53:41.739965 3016 tablet_bootstrap.cc:654] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:41.739964 3015 log.cc:826] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:41.742525 3016 log.cc:826] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:41.744930 3017 tablet_bootstrap.cc:654] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:41.746593 3015 tablet_bootstrap.cc:492] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0: No bootstrap required, opened a new log
I20250815 01:53:41.747275 3015 ts_tablet_manager.cc:1397] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20250815 01:53:41.747326 3017 log.cc:826] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:41.748667 3016 tablet_bootstrap.cc:492] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072: No bootstrap required, opened a new log
I20250815 01:53:41.749306 3016 ts_tablet_manager.cc:1397] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20250815 01:53:41.753053 3017 tablet_bootstrap.cc:492] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177: No bootstrap required, opened a new log
I20250815 01:53:41.753571 3017 ts_tablet_manager.cc:1397] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20250815 01:53:41.769084 3015 raft_consensus.cc:357] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.770220 3015 raft_consensus.cc:383] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:41.770605 3015 raft_consensus.cc:738] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224e0981647c4baba5a6cf2f6b7ca9b0, State: Initialized, Role: FOLLOWER
I20250815 01:53:41.771908 3015 consensus_queue.cc:260] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.776361 3016 raft_consensus.cc:357] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.777391 3016 raft_consensus.cc:383] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:41.777735 3016 raft_consensus.cc:738] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa88fb9f464e429abe650a49dca4f072, State: Initialized, Role: FOLLOWER
I20250815 01:53:41.778885 3016 consensus_queue.cc:260] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.780488 3017 raft_consensus.cc:357] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.781667 3017 raft_consensus.cc:383] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:41.782086 3017 raft_consensus.cc:738] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52eba469cb73439eb8322778ab0c8177, State: Initialized, Role: FOLLOWER
I20250815 01:53:41.783484 3017 consensus_queue.cc:260] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.788703 3015 ts_tablet_manager.cc:1428] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0: Time spent starting tablet: real 0.041s user 0.034s sys 0.000s
I20250815 01:53:41.789384 3016 ts_tablet_manager.cc:1428] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072: Time spent starting tablet: real 0.040s user 0.033s sys 0.004s
I20250815 01:53:41.793498 2996 heartbeater.cc:499] Master 127.0.106.254:34727 was elected leader, sending a full tablet report...
I20250815 01:53:41.794539 3017 ts_tablet_manager.cc:1428] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177: Time spent starting tablet: real 0.041s user 0.030s sys 0.009s
W20250815 01:53:41.815385 2997 tablet.cc:2378] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:41.838865 3021 raft_consensus.cc:491] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:53:41.839334 3021 raft_consensus.cc:513] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.843103 3021 leader_election.cc:290] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52eba469cb73439eb8322778ab0c8177 (127.0.106.195:42821), fa88fb9f464e429abe650a49dca4f072 (127.0.106.193:45207)
I20250815 01:53:41.853484 2951 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7899958bdce4f579e3bef5ba87c6114" candidate_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52eba469cb73439eb8322778ab0c8177" is_pre_election: true
I20250815 01:53:41.853489 2685 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7899958bdce4f579e3bef5ba87c6114" candidate_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa88fb9f464e429abe650a49dca4f072" is_pre_election: true
I20250815 01:53:41.854179 2951 raft_consensus.cc:2466] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 224e0981647c4baba5a6cf2f6b7ca9b0 in term 0.
I20250815 01:53:41.854178 2685 raft_consensus.cc:2466] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 224e0981647c4baba5a6cf2f6b7ca9b0 in term 0.
I20250815 01:53:41.855310 2754 leader_election.cc:304] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 224e0981647c4baba5a6cf2f6b7ca9b0, fa88fb9f464e429abe650a49dca4f072; no voters:
I20250815 01:53:41.856060 3021 raft_consensus.cc:2802] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 01:53:41.856405 3021 raft_consensus.cc:491] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:53:41.856667 3021 raft_consensus.cc:3058] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:41.860915 3021 raft_consensus.cc:513] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.862406 3021 leader_election.cc:290] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [CANDIDATE]: Term 1 election: Requested vote from peers 52eba469cb73439eb8322778ab0c8177 (127.0.106.195:42821), fa88fb9f464e429abe650a49dca4f072 (127.0.106.193:45207)
I20250815 01:53:41.863104 2951 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7899958bdce4f579e3bef5ba87c6114" candidate_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52eba469cb73439eb8322778ab0c8177"
I20250815 01:53:41.863330 2685 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7899958bdce4f579e3bef5ba87c6114" candidate_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa88fb9f464e429abe650a49dca4f072"
I20250815 01:53:41.863503 2951 raft_consensus.cc:3058] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:41.863780 2685 raft_consensus.cc:3058] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:41.867760 2951 raft_consensus.cc:2466] T d7899958bdce4f579e3bef5ba87c6114 P 52eba469cb73439eb8322778ab0c8177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 224e0981647c4baba5a6cf2f6b7ca9b0 in term 1.
I20250815 01:53:41.867995 2685 raft_consensus.cc:2466] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 224e0981647c4baba5a6cf2f6b7ca9b0 in term 1.
I20250815 01:53:41.868726 2752 leader_election.cc:304] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 224e0981647c4baba5a6cf2f6b7ca9b0, 52eba469cb73439eb8322778ab0c8177; no voters:
I20250815 01:53:41.869405 3021 raft_consensus.cc:2802] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:41.871018 3021 raft_consensus.cc:695] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [term 1 LEADER]: Becoming Leader. State: Replica: 224e0981647c4baba5a6cf2f6b7ca9b0, State: Running, Role: LEADER
I20250815 01:53:41.871783 3021 consensus_queue.cc:237] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } }
I20250815 01:53:41.880808 2541 catalog_manager.cc:5582] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 224e0981647c4baba5a6cf2f6b7ca9b0 (127.0.106.194). New cstate: current_term: 1 leader_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "224e0981647c4baba5a6cf2f6b7ca9b0" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41971 } health_report { overall_health: HEALTHY } } }
W20250815 01:53:41.916679 2864 tablet.cc:2378] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:41.935803 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:53:41.938946 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fa88fb9f464e429abe650a49dca4f072 to finish bootstrapping
I20250815 01:53:41.951539 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 224e0981647c4baba5a6cf2f6b7ca9b0 to finish bootstrapping
I20250815 01:53:41.962206 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 52eba469cb73439eb8322778ab0c8177 to finish bootstrapping
W20250815 01:53:42.036242 2731 tablet.cc:2378] T d7899958bdce4f579e3bef5ba87c6114 P fa88fb9f464e429abe650a49dca4f072: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:53:42.402602 3039 consensus_queue.cc:1035] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa88fb9f464e429abe650a49dca4f072" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 45207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:53:42.435386 3043 consensus_queue.cc:1035] T d7899958bdce4f579e3bef5ba87c6114 P 224e0981647c4baba5a6cf2f6b7ca9b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52eba469cb73439eb8322778ab0c8177" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 42821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 01:53:43.663369 2542 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:48320
I20250815 01:53:44.696607 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 2601
I20250815 01:53:44.722783 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 2734
I20250815 01:53:44.750809 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 2867
I20250815 01:53:44.775301 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 2508
2025-08-15T01:53:44Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11830 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250815 01:53:44.833470 427 test_util.cc:276] Using random seed: -1460022878
I20250815 01:53:44.837759 427 ts_itest-base.cc:115] Starting cluster with:
I20250815 01:53:44.837986 427 ts_itest-base.cc:116] --------------
I20250815 01:53:44.838152 427 ts_itest-base.cc:117] 3 tablet servers
I20250815 01:53:44.838294 427 ts_itest-base.cc:118] 3 replicas per TS
I20250815 01:53:44.838531 427 ts_itest-base.cc:119] --------------
2025-08-15T01:53:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T01:53:44Z Disabled control of system clock
I20250815 01:53:44.876579 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:45487
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:45487 with env {}
W20250815 01:53:45.182332 3070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:45.182940 3070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:45.183346 3070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:45.214763 3070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:53:45.215067 3070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:45.215269 3070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:53:45.215464 3070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:53:45.250942 3070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:45487
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:45.252209 3070 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:45.253743 3070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:45.264545 3076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:46.668893 3075 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 3070
W20250815 01:53:47.083379 3075 kernel_stack_watchdog.cc:198] Thread 3070 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:53:45.265189 3077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:47.084185 3070 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.819s user 0.674s sys 1.145s
W20250815 01:53:47.084673 3070 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.820s user 0.674s sys 1.145s
W20250815 01:53:47.086915 3079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:47.088793 3078 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1824 milliseconds
I20250815 01:53:47.088824 3070 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:47.090152 3070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:47.092676 3070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:47.094048 3070 hybrid_clock.cc:648] HybridClock initialized: now 1755222827094010 us; error 65 us; skew 500 ppm
I20250815 01:53:47.094918 3070 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:47.101284 3070 webserver.cc:480] Webserver started at http://127.0.106.254:35169/ using document root <none> and password file <none>
I20250815 01:53:47.102277 3070 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:47.102547 3070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:47.103020 3070 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:47.107671 3070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ed6b44043527428d8b9441d35a58a3b5"
format_stamp: "Formatted at 2025-08-15 01:53:47 on dist-test-slave-h491"
I20250815 01:53:47.108876 3070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ed6b44043527428d8b9441d35a58a3b5"
format_stamp: "Formatted at 2025-08-15 01:53:47 on dist-test-slave-h491"
I20250815 01:53:47.116628 3070 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250815 01:53:47.122742 3086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:47.123976 3070 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 01:53:47.124341 3070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ed6b44043527428d8b9441d35a58a3b5"
format_stamp: "Formatted at 2025-08-15 01:53:47 on dist-test-slave-h491"
I20250815 01:53:47.124696 3070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:47.187711 3070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:47.189236 3070 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:47.189687 3070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:47.260708 3070 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:45487
I20250815 01:53:47.260768 3137 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:45487 every 8 connection(s)
I20250815 01:53:47.263485 3070 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:53:47.266150 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 3070
I20250815 01:53:47.266712 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 01:53:47.270087 3138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:47.294531 3138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap starting.
I20250815 01:53:47.300639 3138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:47.302582 3138 log.cc:826] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:47.307171 3138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: No bootstrap required, opened a new log
I20250815 01:53:47.325467 3138 raft_consensus.cc:357] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:53:47.326143 3138 raft_consensus.cc:383] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:47.326429 3138 raft_consensus.cc:738] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6b44043527428d8b9441d35a58a3b5, State: Initialized, Role: FOLLOWER
I20250815 01:53:47.327105 3138 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:53:47.327621 3138 raft_consensus.cc:397] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:53:47.327893 3138 raft_consensus.cc:491] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:53:47.328222 3138 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:47.332296 3138 raft_consensus.cc:513] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:53:47.333129 3138 leader_election.cc:304] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed6b44043527428d8b9441d35a58a3b5; no voters:
I20250815 01:53:47.335211 3138 leader_election.cc:290] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:53:47.335923 3143 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:47.338086 3143 raft_consensus.cc:695] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 1 LEADER]: Becoming Leader. State: Replica: ed6b44043527428d8b9441d35a58a3b5, State: Running, Role: LEADER
I20250815 01:53:47.338891 3143 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:53:47.339900 3138 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:53:47.348688 3145 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6b44043527428d8b9441d35a58a3b5. Latest consensus state: current_term: 1 leader_uuid: "ed6b44043527428d8b9441d35a58a3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } } }
I20250815 01:53:47.349541 3145 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: This master's current role is: LEADER
I20250815 01:53:47.350822 3144 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed6b44043527428d8b9441d35a58a3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } } }
I20250815 01:53:47.351603 3144 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: This master's current role is: LEADER
I20250815 01:53:47.354200 3152 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:53:47.363454 3152 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:53:47.379281 3152 catalog_manager.cc:1349] Generated new cluster ID: 84c21cc181284ba3849a390a775a93a0
I20250815 01:53:47.379616 3152 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:53:47.397428 3152 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:53:47.399477 3152 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:53:47.418845 3152 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Generated new TSK 0
I20250815 01:53:47.419893 3152 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 01:53:47.442726 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:45487
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 01:53:47.757668 3162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:47.758178 3162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:47.758739 3162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:47.791728 3162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:47.792580 3162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:53:47.828276 3162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:47.829577 3162 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:47.831179 3162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:47.844658 3168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:47.848031 3169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:47.850485 3171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:47.849699 3162 server_base.cc:1047] running on GCE node
I20250815 01:53:49.075536 3162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:49.078370 3162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:49.079979 3162 hybrid_clock.cc:648] HybridClock initialized: now 1755222829079935 us; error 37 us; skew 500 ppm
I20250815 01:53:49.081102 3162 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:49.089041 3162 webserver.cc:480] Webserver started at http://127.0.106.193:39285/ using document root <none> and password file <none>
I20250815 01:53:49.090289 3162 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:49.090601 3162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:49.091203 3162 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:49.097648 3162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f3b8bf9a081a43049b0d43c3471b7a77"
format_stamp: "Formatted at 2025-08-15 01:53:49 on dist-test-slave-h491"
I20250815 01:53:49.099254 3162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f3b8bf9a081a43049b0d43c3471b7a77"
format_stamp: "Formatted at 2025-08-15 01:53:49 on dist-test-slave-h491"
I20250815 01:53:49.108841 3162 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20250815 01:53:49.116508 3178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:49.117898 3162 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250815 01:53:49.118319 3162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f3b8bf9a081a43049b0d43c3471b7a77"
format_stamp: "Formatted at 2025-08-15 01:53:49 on dist-test-slave-h491"
I20250815 01:53:49.118783 3162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:49.194783 3162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:49.196293 3162 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:49.196734 3162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:49.199313 3162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:49.203565 3162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:49.203799 3162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:49.204087 3162 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:49.204249 3162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:49.340389 3162 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:33607
I20250815 01:53:49.340467 3290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:33607 every 8 connection(s)
I20250815 01:53:49.343128 3162 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:53:49.351011 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 3162
I20250815 01:53:49.351435 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 01:53:49.357733 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:45487
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:53:49.370335 3291 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:53:49.370891 3291 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:49.372224 3291 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:53:49.375001 3103 ts_manager.cc:194] Registered new tserver with Master: f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193:33607)
I20250815 01:53:49.376856 3103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:41125
W20250815 01:53:49.672442 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:49.672962 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:49.673488 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:49.705472 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:49.706576 3295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:53:49.741709 3295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:49.743041 3295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:49.744592 3295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:49.756899 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:50.380285 3291 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
W20250815 01:53:49.757414 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:50.956019 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:50.958950 3295 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.202s user 0.381s sys 0.819s
W20250815 01:53:50.959347 3295 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.203s user 0.381s sys 0.819s
W20250815 01:53:50.959383 3303 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1195 milliseconds
I20250815 01:53:50.959645 3295 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:50.960798 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:50.963079 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:50.964440 3295 hybrid_clock.cc:648] HybridClock initialized: now 1755222830964415 us; error 45 us; skew 500 ppm
I20250815 01:53:50.965209 3295 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:50.972339 3295 webserver.cc:480] Webserver started at http://127.0.106.194:42319/ using document root <none> and password file <none>
I20250815 01:53:50.973368 3295 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:50.973680 3295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:50.974133 3295 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:50.978601 3295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ae4e916a450843e1995848bfe38c7e79"
format_stamp: "Formatted at 2025-08-15 01:53:50 on dist-test-slave-h491"
I20250815 01:53:50.979748 3295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ae4e916a450843e1995848bfe38c7e79"
format_stamp: "Formatted at 2025-08-15 01:53:50 on dist-test-slave-h491"
I20250815 01:53:50.987723 3295 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250815 01:53:50.993891 3311 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:50.995175 3295 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250815 01:53:50.995525 3295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ae4e916a450843e1995848bfe38c7e79"
format_stamp: "Formatted at 2025-08-15 01:53:50 on dist-test-slave-h491"
I20250815 01:53:50.995862 3295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:51.071295 3295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:51.072783 3295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:51.073232 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:51.075788 3295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:51.079886 3295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:51.080111 3295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:51.080384 3295 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:51.080551 3295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:51.212441 3295 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:38517
I20250815 01:53:51.212500 3423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:38517 every 8 connection(s)
I20250815 01:53:51.215256 3295 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 01:53:51.219671 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 3295
I20250815 01:53:51.220181 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 01:53:51.227234 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:45487
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:53:51.237509 3424 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:53:51.237952 3424 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:51.239009 3424 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:53:51.241204 3103 ts_manager.cc:194] Registered new tserver with Master: ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517)
I20250815 01:53:51.242573 3103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:60395
W20250815 01:53:51.532794 3428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:51.533339 3428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:51.533845 3428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:51.566929 3428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:51.567992 3428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:53:51.604120 3428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:51.605443 3428 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:51.607048 3428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:51.619448 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:53:52.246079 3424 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
W20250815 01:53:51.620594 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:52.820432 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:52.823163 3436 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1198 milliseconds
I20250815 01:53:52.823261 3428 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:53:52.824491 3428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:53:52.826766 3428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:53:52.828141 3428 hybrid_clock.cc:648] HybridClock initialized: now 1755222832828091 us; error 59 us; skew 500 ppm
I20250815 01:53:52.828965 3428 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:53:52.835107 3428 webserver.cc:480] Webserver started at http://127.0.106.195:33931/ using document root <none> and password file <none>
I20250815 01:53:52.836089 3428 fs_manager.cc:362] Metadata directory not provided
I20250815 01:53:52.836323 3428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:53:52.836796 3428 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:53:52.841334 3428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d80095fdc0a04827843c9b49656f7606"
format_stamp: "Formatted at 2025-08-15 01:53:52 on dist-test-slave-h491"
I20250815 01:53:52.842641 3428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d80095fdc0a04827843c9b49656f7606"
format_stamp: "Formatted at 2025-08-15 01:53:52 on dist-test-slave-h491"
I20250815 01:53:52.850212 3428 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250815 01:53:52.856273 3444 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:52.857481 3428 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250815 01:53:52.857852 3428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d80095fdc0a04827843c9b49656f7606"
format_stamp: "Formatted at 2025-08-15 01:53:52 on dist-test-slave-h491"
I20250815 01:53:52.858229 3428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:53:52.908892 3428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:53:52.910480 3428 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:53:52.910997 3428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:53:52.913473 3428 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:53:52.917675 3428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:53:52.917881 3428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:52.918152 3428 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:53:52.918303 3428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:53:53.061789 3428 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:36135
I20250815 01:53:53.061906 3556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:36135 every 8 connection(s)
I20250815 01:53:53.064366 3428 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 01:53:53.070129 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 3428
I20250815 01:53:53.070689 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 01:53:53.086243 3557 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:53:53.086740 3557 heartbeater.cc:461] Registering TS with master...
I20250815 01:53:53.087770 3557 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:53:53.090216 3103 ts_manager.cc:194] Registered new tserver with Master: d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135)
I20250815 01:53:53.091934 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:53:53.091933 3103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:33943
I20250815 01:53:53.126034 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:53:53.126503 427 test_util.cc:276] Using random seed: -1451729837
I20250815 01:53:53.170037 3103 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53140:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 01:53:53.214548 3226 tablet_service.cc:1468] Processing CreateTablet for tablet 87ce63b5440245848383aca833047788 (DEFAULT_TABLE table=TestTable [id=cf7d5cbc7653486e933ad203ec82c61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:53.216070 3226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87ce63b5440245848383aca833047788. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:53.237025 3577 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap starting.
I20250815 01:53:53.242449 3577 tablet_bootstrap.cc:654] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:53.244257 3577 log.cc:826] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:53.249239 3577 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: No bootstrap required, opened a new log
I20250815 01:53:53.249684 3577 ts_tablet_manager.cc:1397] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20250815 01:53:53.267654 3577 raft_consensus.cc:357] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:53:53.268381 3577 raft_consensus.cc:383] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:53.268679 3577 raft_consensus.cc:738] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Initialized, Role: FOLLOWER
I20250815 01:53:53.269428 3577 consensus_queue.cc:260] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:53:53.269971 3577 raft_consensus.cc:397] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:53:53.270254 3577 raft_consensus.cc:491] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:53:53.270649 3577 raft_consensus.cc:3058] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:53.276085 3577 raft_consensus.cc:513] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:53:53.276846 3577 leader_election.cc:304] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b8bf9a081a43049b0d43c3471b7a77; no voters:
I20250815 01:53:53.278702 3577 leader_election.cc:290] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:53:53.279134 3579 raft_consensus.cc:2802] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:53.282804 3577 ts_tablet_manager.cc:1428] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Time spent starting tablet: real 0.033s user 0.026s sys 0.009s
I20250815 01:53:53.282918 3579 raft_consensus.cc:695] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 1 LEADER]: Becoming Leader. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Running, Role: LEADER
I20250815 01:53:53.283907 3579 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:53:53.298682 3103 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193). New cstate: current_term: 1 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } health_report { overall_health: HEALTHY } } }
I20250815 01:53:53.508924 427 test_util.cc:276] Using random seed: -1451347428
I20250815 01:53:53.532056 3099 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53152:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 01:53:53.563928 3492 tablet_service.cc:1468] Processing CreateTablet for tablet 64f6a349284d446aad7ce94647d9ef7e (DEFAULT_TABLE table=TestTable1 [id=037c39669a2943b1bbfb1547cfce7c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:53.565538 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f6a349284d446aad7ce94647d9ef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:53.585071 3598 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Bootstrap starting.
I20250815 01:53:53.590734 3598 tablet_bootstrap.cc:654] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:53.592438 3598 log.cc:826] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:53.596400 3598 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: No bootstrap required, opened a new log
I20250815 01:53:53.596942 3598 ts_tablet_manager.cc:1397] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250815 01:53:53.615002 3598 raft_consensus.cc:357] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } }
I20250815 01:53:53.615568 3598 raft_consensus.cc:383] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:53.615820 3598 raft_consensus.cc:738] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80095fdc0a04827843c9b49656f7606, State: Initialized, Role: FOLLOWER
I20250815 01:53:53.616518 3598 consensus_queue.cc:260] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } }
I20250815 01:53:53.617027 3598 raft_consensus.cc:397] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:53:53.617305 3598 raft_consensus.cc:491] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:53:53.617625 3598 raft_consensus.cc:3058] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:53.621903 3598 raft_consensus.cc:513] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } }
I20250815 01:53:53.622686 3598 leader_election.cc:304] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80095fdc0a04827843c9b49656f7606; no voters:
I20250815 01:53:53.624666 3598 leader_election.cc:290] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:53:53.625234 3600 raft_consensus.cc:2802] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:53.628078 3557 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
I20250815 01:53:53.629029 3598 ts_tablet_manager.cc:1428] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Time spent starting tablet: real 0.032s user 0.028s sys 0.005s
I20250815 01:53:53.629380 3600 raft_consensus.cc:695] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 1 LEADER]: Becoming Leader. State: Replica: d80095fdc0a04827843c9b49656f7606, State: Running, Role: LEADER
I20250815 01:53:53.630084 3600 consensus_queue.cc:237] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } }
I20250815 01:53:53.641295 3099 catalog_manager.cc:5582] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80095fdc0a04827843c9b49656f7606 (127.0.106.195). New cstate: current_term: 1 leader_uuid: "d80095fdc0a04827843c9b49656f7606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } health_report { overall_health: HEALTHY } } }
I20250815 01:53:53.837095 427 test_util.cc:276] Using random seed: -1451019252
I20250815 01:53:53.859001 3103 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53158:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 01:53:53.887857 3359 tablet_service.cc:1468] Processing CreateTablet for tablet 2cb5bf07f99b478e8157257986a2c575 (DEFAULT_TABLE table=TestTable2 [id=01551f9940ad40d7bf41313fbf7808d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:53:53.889304 3359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cb5bf07f99b478e8157257986a2c575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:53:53.908200 3618 tablet_bootstrap.cc:492] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Bootstrap starting.
I20250815 01:53:53.914004 3618 tablet_bootstrap.cc:654] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Neither blocks nor log segments found. Creating new log.
I20250815 01:53:53.915776 3618 log.cc:826] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Log is configured to *not* fsync() on all Append() calls
I20250815 01:53:53.920187 3618 tablet_bootstrap.cc:492] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: No bootstrap required, opened a new log
I20250815 01:53:53.920655 3618 ts_tablet_manager.cc:1397] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20250815 01:53:53.938987 3618 raft_consensus.cc:357] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:53:53.939749 3618 raft_consensus.cc:383] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:53:53.940030 3618 raft_consensus.cc:738] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Initialized, Role: FOLLOWER
I20250815 01:53:53.940830 3618 consensus_queue.cc:260] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:53:53.941443 3618 raft_consensus.cc:397] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:53:53.941787 3618 raft_consensus.cc:491] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:53:53.942225 3618 raft_consensus.cc:3058] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:53:53.946465 3618 raft_consensus.cc:513] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:53:53.947158 3618 leader_election.cc:304] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4e916a450843e1995848bfe38c7e79; no voters:
I20250815 01:53:53.949143 3618 leader_election.cc:290] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:53:53.949600 3620 raft_consensus.cc:2802] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:53:53.951798 3620 raft_consensus.cc:695] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 1 LEADER]: Becoming Leader. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Running, Role: LEADER
I20250815 01:53:53.952737 3620 consensus_queue.cc:237] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:53:53.953146 3618 ts_tablet_manager.cc:1428] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Time spent starting tablet: real 0.032s user 0.025s sys 0.008s
I20250815 01:53:53.962505 3103 catalog_manager.cc:5582] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4e916a450843e1995848bfe38c7e79 (127.0.106.194). New cstate: current_term: 1 leader_uuid: "ae4e916a450843e1995848bfe38c7e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } health_report { overall_health: HEALTHY } } }
I20250815 01:53:54.202436 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 3070
W20250815 01:53:54.318169 3291 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:45487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:45487: connect: Connection refused (error 111)
W20250815 01:53:54.656119 3557 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:45487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:45487: connect: Connection refused (error 111)
W20250815 01:53:54.980530 3424 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:45487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:45487: connect: Connection refused (error 111)
I20250815 01:53:59.146777 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 3162
I20250815 01:53:59.170166 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 3295
I20250815 01:53:59.196720 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 3428
I20250815 01:53:59.223692 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:45487
--webserver_interface=127.0.106.254
--webserver_port=35169
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:45487 with env {}
W20250815 01:53:59.521850 3698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:53:59.522480 3698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:53:59.522898 3698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:53:59.555609 3698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:53:59.555907 3698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:53:59.556123 3698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:53:59.556322 3698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:53:59.591914 3698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:45487
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=35169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:53:59.593189 3698 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:53:59.594772 3698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:53:59.605255 3704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:59.606822 3705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:53:59.610074 3707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:00.787767 3706 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 01:54:00.787882 3698 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:00.791577 3698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:00.794790 3698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:00.796232 3698 hybrid_clock.cc:648] HybridClock initialized: now 1755222840796187 us; error 65 us; skew 500 ppm
I20250815 01:54:00.797055 3698 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:00.804000 3698 webserver.cc:480] Webserver started at http://127.0.106.254:35169/ using document root <none> and password file <none>
I20250815 01:54:00.805262 3698 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:00.805502 3698 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:00.813544 3698 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250815 01:54:00.818244 3714 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:00.819383 3698 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250815 01:54:00.819728 3698 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ed6b44043527428d8b9441d35a58a3b5"
format_stamp: "Formatted at 2025-08-15 01:53:47 on dist-test-slave-h491"
I20250815 01:54:00.821707 3698 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:00.868359 3698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:00.869840 3698 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:00.870296 3698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:00.938946 3698 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:45487
I20250815 01:54:00.939013 3765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:45487 every 8 connection(s)
I20250815 01:54:00.941778 3698 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:54:00.948655 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 3698
I20250815 01:54:00.950068 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:33607
--local_ip_for_outbound_sockets=127.0.106.193
--tserver_master_addrs=127.0.106.254:45487
--webserver_port=39285
--webserver_interface=127.0.106.193
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:54:00.954759 3766 sys_catalog.cc:263] Verifying existing consensus state
I20250815 01:54:00.959527 3766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap starting.
I20250815 01:54:00.969338 3766 log.cc:826] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:01.017004 3766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:01.017805 3766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap complete.
I20250815 01:54:01.038862 3766 raft_consensus.cc:357] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:01.040956 3766 raft_consensus.cc:738] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6b44043527428d8b9441d35a58a3b5, State: Initialized, Role: FOLLOWER
I20250815 01:54:01.041817 3766 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:01.042307 3766 raft_consensus.cc:397] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:54:01.042608 3766 raft_consensus.cc:491] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:54:01.042910 3766 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 2 FOLLOWER]: Advancing to term 3
I20250815 01:54:01.048534 3766 raft_consensus.cc:513] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:01.049175 3766 leader_election.cc:304] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed6b44043527428d8b9441d35a58a3b5; no voters:
I20250815 01:54:01.051303 3766 leader_election.cc:290] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250815 01:54:01.051702 3770 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 01:54:01.054935 3770 raft_consensus.cc:695] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 3 LEADER]: Becoming Leader. State: Replica: ed6b44043527428d8b9441d35a58a3b5, State: Running, Role: LEADER
I20250815 01:54:01.055929 3770 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:01.056296 3766 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:54:01.068842 3771 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ed6b44043527428d8b9441d35a58a3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } } }
I20250815 01:54:01.069852 3771 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: This master's current role is: LEADER
I20250815 01:54:01.072943 3772 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6b44043527428d8b9441d35a58a3b5. Latest consensus state: current_term: 3 leader_uuid: "ed6b44043527428d8b9441d35a58a3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } } }
I20250815 01:54:01.075129 3772 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: This master's current role is: LEADER
I20250815 01:54:01.088780 3776 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:54:01.103701 3776 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=01551f9940ad40d7bf41313fbf7808d8]
I20250815 01:54:01.106092 3776 catalog_manager.cc:671] Loaded metadata for table TestTable [id=74eed1774f37458b851316eb057f7c2a]
I20250815 01:54:01.108310 3776 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=9ee2f5eb83b34e1cbf2f82fceb9e5f5f]
I20250815 01:54:01.117234 3776 tablet_loader.cc:96] loaded metadata for tablet 2cb5bf07f99b478e8157257986a2c575 (table TestTable2 [id=01551f9940ad40d7bf41313fbf7808d8])
I20250815 01:54:01.118621 3776 tablet_loader.cc:96] loaded metadata for tablet 64f6a349284d446aad7ce94647d9ef7e (table TestTable1 [id=9ee2f5eb83b34e1cbf2f82fceb9e5f5f])
I20250815 01:54:01.119879 3776 tablet_loader.cc:96] loaded metadata for tablet 87ce63b5440245848383aca833047788 (table TestTable [id=74eed1774f37458b851316eb057f7c2a])
I20250815 01:54:01.121192 3776 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:54:01.127578 3776 catalog_manager.cc:1261] Loaded cluster ID: 84c21cc181284ba3849a390a775a93a0
I20250815 01:54:01.127977 3776 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:54:01.136503 3776 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:54:01.141860 3776 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Loaded TSK: 0
I20250815 01:54:01.143697 3776 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 01:54:01.305230 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:01.305888 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:01.306522 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:01.338717 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:01.339577 3768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:54:01.375414 3768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:33607
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=39285
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:01.376725 3768 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:01.378291 3768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:01.391741 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:01.395539 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:01.396284 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:01.396273 3768 server_base.cc:1047] running on GCE node
I20250815 01:54:02.561833 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:02.564697 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:02.566260 3768 hybrid_clock.cc:648] HybridClock initialized: now 1755222842566210 us; error 50 us; skew 500 ppm
I20250815 01:54:02.567361 3768 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:02.574934 3768 webserver.cc:480] Webserver started at http://127.0.106.193:39285/ using document root <none> and password file <none>
I20250815 01:54:02.576193 3768 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:02.576494 3768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:02.587397 3768 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250815 01:54:02.593530 3804 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:02.594818 3768 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20250815 01:54:02.595222 3768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f3b8bf9a081a43049b0d43c3471b7a77"
format_stamp: "Formatted at 2025-08-15 01:53:49 on dist-test-slave-h491"
I20250815 01:54:02.597937 3768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:02.671259 3768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:02.673180 3768 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:02.673748 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:02.676985 3768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:02.684556 3811 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 01:54:02.692239 3768 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 01:54:02.692615 3768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250815 01:54:02.692941 3768 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 01:54:02.700253 3768 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 01:54:02.700546 3768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20250815 01:54:02.700947 3811 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap starting.
I20250815 01:54:02.773788 3811 log.cc:826] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:02.893164 3811 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:02.894264 3811 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap complete.
I20250815 01:54:02.896066 3811 ts_tablet_manager.cc:1397] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Time spent bootstrapping tablet: real 0.196s user 0.146s sys 0.044s
I20250815 01:54:02.912676 3768 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:33607
I20250815 01:54:02.912848 3918 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:33607 every 8 connection(s)
I20250815 01:54:02.915293 3768 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:54:02.923499 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 3768
I20250815 01:54:02.925855 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:38517
--local_ip_for_outbound_sockets=127.0.106.194
--tserver_master_addrs=127.0.106.254:45487
--webserver_port=42319
--webserver_interface=127.0.106.194
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:54:02.917536 3811 raft_consensus.cc:357] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:54:02.927584 3811 raft_consensus.cc:738] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Initialized, Role: FOLLOWER
I20250815 01:54:02.929456 3811 consensus_queue.cc:260] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:54:02.931034 3811 raft_consensus.cc:397] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:54:02.931730 3811 raft_consensus.cc:491] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:54:02.932472 3811 raft_consensus.cc:3058] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:54:02.952591 3811 raft_consensus.cc:513] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:54:02.953280 3811 leader_election.cc:304] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b8bf9a081a43049b0d43c3471b7a77; no voters:
I20250815 01:54:02.969043 3811 leader_election.cc:290] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 01:54:02.969904 3924 raft_consensus.cc:2802] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 01:54:02.981670 3919 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:54:02.982331 3919 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:02.983453 3924 raft_consensus.cc:695] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEADER]: Becoming Leader. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Running, Role: LEADER
I20250815 01:54:02.983762 3919 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:54:02.984339 3924 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:54:02.994904 3811 ts_tablet_manager.cc:1428] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Time spent starting tablet: real 0.099s user 0.038s sys 0.028s
I20250815 01:54:02.998751 3731 ts_manager.cc:194] Registered new tserver with Master: f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193:33607)
I20250815 01:54:03.001791 3731 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: term changed from 0 to 2, leader changed from <none> to f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193), VOTER f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193) added. New cstate: current_term: 2 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } health_report { overall_health: HEALTHY } } }
W20250815 01:54:03.053537 3731 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 87ce63b5440245848383aca833047788 with cas_config_opid_index -1: no extra replica candidate found for tablet 87ce63b5440245848383aca833047788 (table TestTable [id=74eed1774f37458b851316eb057f7c2a]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250815 01:54:03.055917 3731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:59685
I20250815 01:54:03.059986 3919 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
W20250815 01:54:03.295909 3923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:03.296406 3923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:03.296895 3923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:03.328769 3923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:03.329631 3923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:54:03.364929 3923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:38517
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=42319
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:03.366261 3923 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:03.367995 3923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:03.379897 3938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:03.380511 3939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:04.947360 3923 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.567s user 0.553s sys 0.997s
W20250815 01:54:04.784417 3937 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3923
W20250815 01:54:04.949324 3941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:04.949503 3923 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.569s user 0.553s sys 0.998s
W20250815 01:54:04.953858 3940 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1573 milliseconds
I20250815 01:54:04.953904 3923 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:04.955355 3923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:04.957808 3923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:04.959379 3923 hybrid_clock.cc:648] HybridClock initialized: now 1755222844959325 us; error 59 us; skew 500 ppm
I20250815 01:54:04.960430 3923 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:04.967757 3923 webserver.cc:480] Webserver started at http://127.0.106.194:42319/ using document root <none> and password file <none>
I20250815 01:54:04.968958 3923 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:04.969260 3923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:04.978480 3923 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.002s
I20250815 01:54:04.982983 3949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:04.984154 3923 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250815 01:54:04.984496 3923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ae4e916a450843e1995848bfe38c7e79"
format_stamp: "Formatted at 2025-08-15 01:53:50 on dist-test-slave-h491"
I20250815 01:54:04.986479 3923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:05.044765 3923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:05.046231 3923 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:05.046710 3923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:05.049186 3923 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:05.054709 3956 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 01:54:05.062177 3923 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 01:54:05.062469 3923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250815 01:54:05.062747 3923 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 01:54:05.067499 3923 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 01:54:05.067732 3923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.002s
I20250815 01:54:05.068132 3956 tablet_bootstrap.cc:492] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Bootstrap starting.
I20250815 01:54:05.126495 3956 log.cc:826] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:05.219765 3956 tablet_bootstrap.cc:492] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:05.220650 3956 tablet_bootstrap.cc:492] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Bootstrap complete.
I20250815 01:54:05.221940 3956 ts_tablet_manager.cc:1397] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Time spent bootstrapping tablet: real 0.154s user 0.113s sys 0.036s
I20250815 01:54:05.238747 3956 raft_consensus.cc:357] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:54:05.241395 3956 raft_consensus.cc:738] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Initialized, Role: FOLLOWER
I20250815 01:54:05.242233 3923 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:38517
I20250815 01:54:05.242596 4063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:38517 every 8 connection(s)
I20250815 01:54:05.242484 3956 consensus_queue.cc:260] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:54:05.243157 3956 raft_consensus.cc:397] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:54:05.243413 3956 raft_consensus.cc:491] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:54:05.243721 3956 raft_consensus.cc:3058] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:54:05.244866 3923 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 01:54:05.250385 3956 raft_consensus.cc:513] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:54:05.251421 3956 leader_election.cc:304] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4e916a450843e1995848bfe38c7e79; no voters:
I20250815 01:54:05.253587 3956 leader_election.cc:290] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 01:54:05.254005 4068 raft_consensus.cc:2802] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 01:54:05.255203 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 3923
I20250815 01:54:05.257405 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:36135
--local_ip_for_outbound_sockets=127.0.106.195
--tserver_master_addrs=127.0.106.254:45487
--webserver_port=33931
--webserver_interface=127.0.106.195
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:54:05.270103 4064 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:54:05.270601 4064 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:05.271340 3956 ts_tablet_manager.cc:1428] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Time spent starting tablet: real 0.049s user 0.036s sys 0.015s
I20250815 01:54:05.271611 4064 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:54:05.274807 4068 raft_consensus.cc:695] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 2 LEADER]: Becoming Leader. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Running, Role: LEADER
I20250815 01:54:05.275759 4068 consensus_queue.cc:237] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:54:05.287736 3731 ts_manager.cc:194] Registered new tserver with Master: ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517)
I20250815 01:54:05.288740 3731 catalog_manager.cc:5582] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "ae4e916a450843e1995848bfe38c7e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } health_report { overall_health: HEALTHY } } }
I20250815 01:54:05.299597 3731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:34519
I20250815 01:54:05.303449 4064 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
I20250815 01:54:05.370216 3874 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:05.377436 4082 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } } }
I20250815 01:54:05.390018 3715 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 87ce63b5440245848383aca833047788 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250815 01:54:05.395625 3731 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: config changed from index -1 to 9, NON_VOTER ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) added. New cstate: current_term: 2 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250815 01:54:05.405413 3808 consensus_peers.cc:489] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 -> Peer ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517): Couldn't send request to peer ae4e916a450843e1995848bfe38c7e79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87ce63b5440245848383aca833047788. This is attempt 1: this message will repeat every 5th retry.
W20250815 01:54:05.589591 4071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:05.590102 4071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:05.590643 4071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:05.622130 4071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:05.623050 4071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:54:05.658419 4071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:36135
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=33931
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:05.659734 4071 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:05.661317 4071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:05.672850 4090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:05.866498 4096 ts_tablet_manager.cc:927] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Initiating tablet copy from peer f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193:33607)
I20250815 01:54:05.879408 4096 tablet_copy_client.cc:323] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.193:33607
I20250815 01:54:05.922295 3894 tablet_copy_service.cc:140] P f3b8bf9a081a43049b0d43c3471b7a77: Received BeginTabletCopySession request for tablet 87ce63b5440245848383aca833047788 from peer ae4e916a450843e1995848bfe38c7e79 ({username='slave'} at 127.0.106.194:44625)
I20250815 01:54:05.923159 3894 tablet_copy_service.cc:161] P f3b8bf9a081a43049b0d43c3471b7a77: Beginning new tablet copy session on tablet 87ce63b5440245848383aca833047788 from peer ae4e916a450843e1995848bfe38c7e79 at {username='slave'} at 127.0.106.194:44625: session id = ae4e916a450843e1995848bfe38c7e79-87ce63b5440245848383aca833047788
I20250815 01:54:05.933607 3894 tablet_copy_source_session.cc:215] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 01:54:05.960291 4096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87ce63b5440245848383aca833047788. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:54:06.022300 4096 tablet_copy_client.cc:806] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: tablet copy: Starting download of 0 data blocks...
I20250815 01:54:06.023329 4096 tablet_copy_client.cc:670] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: tablet copy: Starting download of 1 WAL segments...
I20250815 01:54:06.029603 4096 tablet_copy_client.cc:538] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 01:54:06.048836 4096 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Bootstrap starting.
I20250815 01:54:06.302431 4096 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:06.303646 4096 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Bootstrap complete.
I20250815 01:54:06.304548 4096 ts_tablet_manager.cc:1397] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Time spent bootstrapping tablet: real 0.256s user 0.163s sys 0.030s
I20250815 01:54:06.308043 4096 raft_consensus.cc:357] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:06.308737 4096 raft_consensus.cc:738] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Initialized, Role: LEARNER
I20250815 01:54:06.309280 4096 consensus_queue.cc:260] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:06.329357 4096 ts_tablet_manager.cc:1428] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Time spent starting tablet: real 0.024s user 0.005s sys 0.004s
I20250815 01:54:06.334717 3894 tablet_copy_service.cc:342] P f3b8bf9a081a43049b0d43c3471b7a77: Request end of tablet copy session ae4e916a450843e1995848bfe38c7e79-87ce63b5440245848383aca833047788 received from {username='slave'} at 127.0.106.194:44625
I20250815 01:54:06.335223 3894 tablet_copy_service.cc:434] P f3b8bf9a081a43049b0d43c3471b7a77: ending tablet copy session ae4e916a450843e1995848bfe38c7e79-87ce63b5440245848383aca833047788 on tablet 87ce63b5440245848383aca833047788 with peer ae4e916a450843e1995848bfe38c7e79
I20250815 01:54:06.433907 4019 raft_consensus.cc:1215] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250815 01:54:07.025413 4105 raft_consensus.cc:1062] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: attempting to promote NON_VOTER ae4e916a450843e1995848bfe38c7e79 to VOTER
I20250815 01:54:07.027297 4105 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:07.057085 4019 raft_consensus.cc:1273] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Refusing update from remote peer f3b8bf9a081a43049b0d43c3471b7a77: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250815 01:54:07.059595 4106 consensus_queue.cc:1035] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250815 01:54:07.074397 4106 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } }
I20250815 01:54:07.081316 4018 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } }
I20250815 01:54:07.106963 3730 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: config changed from index 9 to 10, ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250815 01:54:07.078871 4089 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4071
W20250815 01:54:07.519520 4071 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.846s user 0.667s sys 1.113s
W20250815 01:54:07.519932 4071 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.846s user 0.667s sys 1.113s
W20250815 01:54:05.674769 4091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:07.522070 4093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:07.525259 4092 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1846 milliseconds
I20250815 01:54:07.525314 4071 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:07.526577 4071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:07.528664 4071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:07.530081 4071 hybrid_clock.cc:648] HybridClock initialized: now 1755222847530049 us; error 45 us; skew 500 ppm
I20250815 01:54:07.530929 4071 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:07.537266 4071 webserver.cc:480] Webserver started at http://127.0.106.195:33931/ using document root <none> and password file <none>
I20250815 01:54:07.538230 4071 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:07.538515 4071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:07.546496 4071 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.006s
I20250815 01:54:07.551209 4118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:07.552341 4071 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 01:54:07.552655 4071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d80095fdc0a04827843c9b49656f7606"
format_stamp: "Formatted at 2025-08-15 01:53:52 on dist-test-slave-h491"
I20250815 01:54:07.554646 4071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:07.612439 4071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:07.613973 4071 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:07.614461 4071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:07.616981 4071 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:07.622756 4125 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 01:54:07.624514 3717 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 87ce63b5440245848383aca833047788 with cas_config_opid_index 9: aborting the task: latest config opid_index 10; task opid_index 9
I20250815 01:54:07.631050 4071 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 01:54:07.631314 4071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250815 01:54:07.631588 4071 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 01:54:07.636452 4071 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 01:54:07.636654 4071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.001s
I20250815 01:54:07.637020 4125 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Bootstrap starting.
I20250815 01:54:07.693145 4125 log.cc:826] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:07.777532 4125 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:07.778651 4125 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Bootstrap complete.
I20250815 01:54:07.780314 4125 ts_tablet_manager.cc:1397] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Time spent bootstrapping tablet: real 0.144s user 0.100s sys 0.040s
I20250815 01:54:07.798107 4125 raft_consensus.cc:357] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } }
I20250815 01:54:07.800642 4125 raft_consensus.cc:738] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80095fdc0a04827843c9b49656f7606, State: Initialized, Role: FOLLOWER
I20250815 01:54:07.801801 4125 consensus_queue.cc:260] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } }
I20250815 01:54:07.802722 4125 raft_consensus.cc:397] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:54:07.803081 4125 raft_consensus.cc:491] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:54:07.803566 4125 raft_consensus.cc:3058] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:54:07.810079 4125 raft_consensus.cc:513] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } }
I20250815 01:54:07.810947 4125 leader_election.cc:304] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80095fdc0a04827843c9b49656f7606; no voters:
I20250815 01:54:07.814395 4125 leader_election.cc:290] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 01:54:07.814851 4227 raft_consensus.cc:2802] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 01:54:07.818312 4227 raft_consensus.cc:695] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 LEADER]: Becoming Leader. State: Replica: d80095fdc0a04827843c9b49656f7606, State: Running, Role: LEADER
I20250815 01:54:07.819630 4227 consensus_queue.cc:237] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } }
I20250815 01:54:07.824113 4125 ts_tablet_manager.cc:1428] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Time spent starting tablet: real 0.044s user 0.042s sys 0.001s
I20250815 01:54:07.832021 4071 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:36135
I20250815 01:54:07.832432 4236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:36135 every 8 connection(s)
I20250815 01:54:07.834817 4071 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 01:54:07.841233 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 4071
I20250815 01:54:07.855173 4238 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:54:07.855728 4238 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:07.856850 4238 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:54:07.860234 3730 ts_manager.cc:194] Registered new tserver with Master: d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135)
I20250815 01:54:07.861129 3730 catalog_manager.cc:5582] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 reported cstate change: term changed from 0 to 2, leader changed from <none> to d80095fdc0a04827843c9b49656f7606 (127.0.106.195), VOTER d80095fdc0a04827843c9b49656f7606 (127.0.106.195) added. New cstate: current_term: 2 leader_uuid: "d80095fdc0a04827843c9b49656f7606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } health_report { overall_health: HEALTHY } } }
I20250815 01:54:07.872632 3730 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:57101
I20250815 01:54:07.876451 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:54:07.877161 4238 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
I20250815 01:54:07.881366 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250815 01:54:07.885401 427 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250815 01:54:07.887439 4188 consensus_queue.cc:237] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } }
I20250815 01:54:07.890854 4229 raft_consensus.cc:2953] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } } }
I20250815 01:54:07.899398 3715 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 64f6a349284d446aad7ce94647d9ef7e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250815 01:54:07.901918 4119 consensus_peers.cc:489] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 -> Peer f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193:33607): Couldn't send request to peer f3b8bf9a081a43049b0d43c3471b7a77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64f6a349284d446aad7ce94647d9ef7e. This is attempt 1: this message will repeat every 5th retry.
I20250815 01:54:07.902760 3731 catalog_manager.cc:5582] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 reported cstate change: config changed from index -1 to 8, NON_VOTER f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193) added. New cstate: current_term: 2 leader_uuid: "d80095fdc0a04827843c9b49656f7606" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 01:54:07.913049 4188 consensus_queue.cc:237] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:07.916905 4227 raft_consensus.cc:2953] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } } }
W20250815 01:54:07.918766 4119 consensus_peers.cc:489] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 -> Peer f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193:33607): Couldn't send request to peer f3b8bf9a081a43049b0d43c3471b7a77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64f6a349284d446aad7ce94647d9ef7e. This is attempt 1: this message will repeat every 5th retry.
I20250815 01:54:07.925556 3715 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 64f6a349284d446aad7ce94647d9ef7e with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250815 01:54:07.928247 3731 catalog_manager.cc:5582] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 reported cstate change: config changed from index 8 to 9, NON_VOTER ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) added. New cstate: current_term: 2 leader_uuid: "d80095fdc0a04827843c9b49656f7606" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250815 01:54:07.930281 4122 consensus_peers.cc:489] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 -> Peer ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517): Couldn't send request to peer ae4e916a450843e1995848bfe38c7e79. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64f6a349284d446aad7ce94647d9ef7e. This is attempt 1: this message will repeat every 5th retry.
I20250815 01:54:08.301545 3874 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: true } }
I20250815 01:54:08.307348 4018 raft_consensus.cc:1273] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Refusing update from remote peer f3b8bf9a081a43049b0d43c3471b7a77: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 01:54:08.308486 4106 consensus_queue.cc:1035] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 01:54:08.314582 4252 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER d80095fdc0a04827843c9b49656f7606 (127.0.106.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: true } } }
W20250815 01:54:08.315707 3805 consensus_peers.cc:489] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 -> Peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Couldn't send request to peer d80095fdc0a04827843c9b49656f7606. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87ce63b5440245848383aca833047788. This is attempt 1: this message will repeat every 5th retry.
I20250815 01:54:08.316327 4018 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER d80095fdc0a04827843c9b49656f7606 (127.0.106.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: true } } }
I20250815 01:54:08.321101 3715 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 87ce63b5440245848383aca833047788 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 7)
I20250815 01:54:08.324592 3731 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: config changed from index 10 to 11, NON_VOTER d80095fdc0a04827843c9b49656f7606 (127.0.106.195) added. New cstate: current_term: 2 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 01:54:08.347126 4256 ts_tablet_manager.cc:927] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Initiating tablet copy from peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135)
I20250815 01:54:08.348594 4256 tablet_copy_client.cc:323] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.195:36135
I20250815 01:54:08.358981 4208 tablet_copy_service.cc:140] P d80095fdc0a04827843c9b49656f7606: Received BeginTabletCopySession request for tablet 64f6a349284d446aad7ce94647d9ef7e from peer ae4e916a450843e1995848bfe38c7e79 ({username='slave'} at 127.0.106.194:51083)
I20250815 01:54:08.359504 4208 tablet_copy_service.cc:161] P d80095fdc0a04827843c9b49656f7606: Beginning new tablet copy session on tablet 64f6a349284d446aad7ce94647d9ef7e from peer ae4e916a450843e1995848bfe38c7e79 at {username='slave'} at 127.0.106.194:51083: session id = ae4e916a450843e1995848bfe38c7e79-64f6a349284d446aad7ce94647d9ef7e
I20250815 01:54:08.364099 4208 tablet_copy_source_session.cc:215] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 01:54:08.367224 4256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f6a349284d446aad7ce94647d9ef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:54:08.370189 4259 ts_tablet_manager.cc:927] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Initiating tablet copy from peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135)
I20250815 01:54:08.372522 4259 tablet_copy_client.cc:323] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.195:36135
I20250815 01:54:08.374085 4208 tablet_copy_service.cc:140] P d80095fdc0a04827843c9b49656f7606: Received BeginTabletCopySession request for tablet 64f6a349284d446aad7ce94647d9ef7e from peer f3b8bf9a081a43049b0d43c3471b7a77 ({username='slave'} at 127.0.106.193:51695)
I20250815 01:54:08.374511 4208 tablet_copy_service.cc:161] P d80095fdc0a04827843c9b49656f7606: Beginning new tablet copy session on tablet 64f6a349284d446aad7ce94647d9ef7e from peer f3b8bf9a081a43049b0d43c3471b7a77 at {username='slave'} at 127.0.106.193:51695: session id = f3b8bf9a081a43049b0d43c3471b7a77-64f6a349284d446aad7ce94647d9ef7e
I20250815 01:54:08.378607 4208 tablet_copy_source_session.cc:215] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 01:54:08.379159 4256 tablet_copy_client.cc:806] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: tablet copy: Starting download of 0 data blocks...
I20250815 01:54:08.379666 4256 tablet_copy_client.cc:670] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: tablet copy: Starting download of 1 WAL segments...
I20250815 01:54:08.381603 4259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64f6a349284d446aad7ce94647d9ef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:54:08.383518 4256 tablet_copy_client.cc:538] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 01:54:08.392047 4256 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Bootstrap starting.
I20250815 01:54:08.397464 4259 tablet_copy_client.cc:806] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: tablet copy: Starting download of 0 data blocks...
I20250815 01:54:08.398037 4259 tablet_copy_client.cc:670] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: tablet copy: Starting download of 1 WAL segments...
I20250815 01:54:08.402042 4259 tablet_copy_client.cc:538] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 01:54:08.410917 4259 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap starting.
I20250815 01:54:08.497269 4256 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:08.497941 4256 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Bootstrap complete.
I20250815 01:54:08.498445 4256 ts_tablet_manager.cc:1397] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Time spent bootstrapping tablet: real 0.107s user 0.099s sys 0.008s
I20250815 01:54:08.499871 4256 raft_consensus.cc:357] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:08.500331 4256 raft_consensus.cc:738] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Initialized, Role: LEARNER
I20250815 01:54:08.500761 4256 consensus_queue.cc:260] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:08.502224 4256 ts_tablet_manager.cc:1428] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20250815 01:54:08.503903 4208 tablet_copy_service.cc:342] P d80095fdc0a04827843c9b49656f7606: Request end of tablet copy session ae4e916a450843e1995848bfe38c7e79-64f6a349284d446aad7ce94647d9ef7e received from {username='slave'} at 127.0.106.194:51083
I20250815 01:54:08.504338 4208 tablet_copy_service.cc:434] P d80095fdc0a04827843c9b49656f7606: ending tablet copy session ae4e916a450843e1995848bfe38c7e79-64f6a349284d446aad7ce94647d9ef7e on tablet 64f6a349284d446aad7ce94647d9ef7e with peer ae4e916a450843e1995848bfe38c7e79
I20250815 01:54:08.511435 4259 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:08.512111 4259 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap complete.
I20250815 01:54:08.512535 4259 ts_tablet_manager.cc:1397] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Time spent bootstrapping tablet: real 0.102s user 0.092s sys 0.008s
I20250815 01:54:08.514117 4259 raft_consensus.cc:357] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:08.514657 4259 raft_consensus.cc:738] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Initialized, Role: LEARNER
I20250815 01:54:08.515194 4259 consensus_queue.cc:260] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: true } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:08.516796 4259 ts_tablet_manager.cc:1428] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250815 01:54:08.518435 4208 tablet_copy_service.cc:342] P d80095fdc0a04827843c9b49656f7606: Request end of tablet copy session f3b8bf9a081a43049b0d43c3471b7a77-64f6a349284d446aad7ce94647d9ef7e received from {username='slave'} at 127.0.106.193:51695
I20250815 01:54:08.518725 4208 tablet_copy_service.cc:434] P d80095fdc0a04827843c9b49656f7606: ending tablet copy session f3b8bf9a081a43049b0d43c3471b7a77-64f6a349284d446aad7ce94647d9ef7e on tablet 64f6a349284d446aad7ce94647d9ef7e with peer f3b8bf9a081a43049b0d43c3471b7a77
I20250815 01:54:08.783758 4263 ts_tablet_manager.cc:927] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Initiating tablet copy from peer f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193:33607)
I20250815 01:54:08.785612 4263 tablet_copy_client.cc:323] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.193:33607
I20250815 01:54:08.787148 3894 tablet_copy_service.cc:140] P f3b8bf9a081a43049b0d43c3471b7a77: Received BeginTabletCopySession request for tablet 87ce63b5440245848383aca833047788 from peer d80095fdc0a04827843c9b49656f7606 ({username='slave'} at 127.0.106.195:60531)
I20250815 01:54:08.787564 3894 tablet_copy_service.cc:161] P f3b8bf9a081a43049b0d43c3471b7a77: Beginning new tablet copy session on tablet 87ce63b5440245848383aca833047788 from peer d80095fdc0a04827843c9b49656f7606 at {username='slave'} at 127.0.106.195:60531: session id = d80095fdc0a04827843c9b49656f7606-87ce63b5440245848383aca833047788
I20250815 01:54:08.791785 3894 tablet_copy_source_session.cc:215] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 01:54:08.794884 4263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87ce63b5440245848383aca833047788. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:54:08.798926 3874 raft_consensus.cc:1215] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250815 01:54:08.808429 4018 raft_consensus.cc:1215] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250815 01:54:08.813918 4263 tablet_copy_client.cc:806] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: tablet copy: Starting download of 0 data blocks...
I20250815 01:54:08.814504 4263 tablet_copy_client.cc:670] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: tablet copy: Starting download of 1 WAL segments...
I20250815 01:54:08.817866 4263 tablet_copy_client.cc:538] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 01:54:08.823282 4263 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Bootstrap starting.
I20250815 01:54:08.890426 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f3b8bf9a081a43049b0d43c3471b7a77 to finish bootstrapping
I20250815 01:54:08.913036 4263 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:08.914019 4263 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Bootstrap complete.
I20250815 01:54:08.914695 4263 ts_tablet_manager.cc:1397] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Time spent bootstrapping tablet: real 0.092s user 0.083s sys 0.008s
I20250815 01:54:08.917138 4263 raft_consensus.cc:357] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: true } }
I20250815 01:54:08.917618 4263 raft_consensus.cc:738] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d80095fdc0a04827843c9b49656f7606, State: Initialized, Role: LEARNER
I20250815 01:54:08.918028 4263 consensus_queue.cc:260] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: true } }
I20250815 01:54:08.919772 4263 ts_tablet_manager.cc:1428] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250815 01:54:08.920109 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ae4e916a450843e1995848bfe38c7e79 to finish bootstrapping
I20250815 01:54:08.921480 3894 tablet_copy_service.cc:342] P f3b8bf9a081a43049b0d43c3471b7a77: Request end of tablet copy session d80095fdc0a04827843c9b49656f7606-87ce63b5440245848383aca833047788 received from {username='slave'} at 127.0.106.195:60531
I20250815 01:54:08.921895 3894 tablet_copy_service.cc:434] P f3b8bf9a081a43049b0d43c3471b7a77: ending tablet copy session d80095fdc0a04827843c9b49656f7606-87ce63b5440245848383aca833047788 on tablet 87ce63b5440245848383aca833047788 with peer d80095fdc0a04827843c9b49656f7606
I20250815 01:54:08.942740 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d80095fdc0a04827843c9b49656f7606 to finish bootstrapping
I20250815 01:54:09.180856 4188 raft_consensus.cc:1215] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250815 01:54:09.224992 4168 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 01:54:09.228071 3999 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 01:54:09.232350 3854 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 01:54:09.349131 4266 raft_consensus.cc:1062] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: attempting to promote NON_VOTER f3b8bf9a081a43049b0d43c3471b7a77 to VOTER
I20250815 01:54:09.351917 4266 consensus_queue.cc:237] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } }
I20250815 01:54:09.377485 4018 raft_consensus.cc:1273] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Refusing update from remote peer d80095fdc0a04827843c9b49656f7606: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250815 01:54:09.379119 3874 raft_consensus.cc:1273] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEARNER]: Refusing update from remote peer d80095fdc0a04827843c9b49656f7606: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250815 01:54:09.381103 4299 consensus_queue.cc:1035] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250815 01:54:09.382553 4299 consensus_queue.cc:1035] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250815 01:54:09.412796 4299 raft_consensus.cc:1025] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 LEADER]: attempt to promote peer ae4e916a450843e1995848bfe38c7e79: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250815 01:54:09.415638 4266 raft_consensus.cc:2953] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } } }
I20250815 01:54:09.424610 3873 raft_consensus.cc:2953] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } } }
I20250815 01:54:09.430528 4018 raft_consensus.cc:2953] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } } }
I20250815 01:54:09.438529 3729 catalog_manager.cc:5582] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: config changed from index 9 to 10, f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d80095fdc0a04827843c9b49656f7606" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: true } } }
I20250815 01:54:09.459667 4266 raft_consensus.cc:1062] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: attempting to promote NON_VOTER ae4e916a450843e1995848bfe38c7e79 to VOTER
I20250815 01:54:09.461587 4266 consensus_queue.cc:237] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:09.468156 4018 raft_consensus.cc:1273] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 LEARNER]: Refusing update from remote peer d80095fdc0a04827843c9b49656f7606: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 01:54:09.469774 4299 consensus_queue.cc:1035] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 01:54:09.476394 3873 raft_consensus.cc:1273] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Refusing update from remote peer d80095fdc0a04827843c9b49656f7606: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 01:54:09.478029 4299 consensus_queue.cc:1035] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 01:54:09.483589 4301 raft_consensus.cc:2953] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } }
I20250815 01:54:09.488605 4018 raft_consensus.cc:2953] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } }
I20250815 01:54:09.489360 3873 raft_consensus.cc:2953] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } }
I20250815 01:54:09.537926 3728 catalog_manager.cc:5582] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: config changed from index 10 to 11, ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d80095fdc0a04827843c9b49656f7606" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ed6b44043527428d8b9441d35a58a3b5 | 127.0.106.254:45487 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:42155 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
ae4e916a450843e1995848bfe38c7e79 | 127.0.106.194:38517 | HEALTHY | <none> | 1 | 0
d80095fdc0a04827843c9b49656f7606 | 127.0.106.195:36135 | HEALTHY | <none> | 1 | 0
f3b8bf9a081a43049b0d43c3471b7a77 | 127.0.106.193:33607 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.193 | experimental | 127.0.106.193:33607
local_ip_for_outbound_sockets | 127.0.106.194 | experimental | 127.0.106.194:38517
local_ip_for_outbound_sockets | 127.0.106.195 | experimental | 127.0.106.195:36135
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.106.193:33607
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.106.194:38517
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.106.195:36135
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:42155 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 01:54:09.557942 427 log_verifier.cc:126] Checking tablet 2cb5bf07f99b478e8157257986a2c575
I20250815 01:54:09.590468 427 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2cb5bf07f99b478e8157257986a2c575
I20250815 01:54:09.590768 427 log_verifier.cc:126] Checking tablet 64f6a349284d446aad7ce94647d9ef7e
I20250815 01:54:09.613144 4293 raft_consensus.cc:1062] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: attempting to promote NON_VOTER d80095fdc0a04827843c9b49656f7606 to VOTER
I20250815 01:54:09.614709 4293 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:09.619693 4018 raft_consensus.cc:1273] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Refusing update from remote peer f3b8bf9a081a43049b0d43c3471b7a77: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 01:54:09.619942 4188 raft_consensus.cc:1273] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 LEARNER]: Refusing update from remote peer f3b8bf9a081a43049b0d43c3471b7a77: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 01:54:09.620812 4293 consensus_queue.cc:1035] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250815 01:54:09.621625 4106 consensus_queue.cc:1035] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250815 01:54:09.632277 4018 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d80095fdc0a04827843c9b49656f7606 (127.0.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } } }
I20250815 01:54:09.630259 4293 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d80095fdc0a04827843c9b49656f7606 (127.0.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } } }
I20250815 01:54:09.639483 4187 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d80095fdc0a04827843c9b49656f7606 (127.0.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } } }
I20250815 01:54:09.646984 3729 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 reported cstate change: config changed from index 11 to 12, d80095fdc0a04827843c9b49656f7606 (127.0.106.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } } }
I20250815 01:54:09.689795 427 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 64f6a349284d446aad7ce94647d9ef7e
I20250815 01:54:09.690063 427 log_verifier.cc:126] Checking tablet 87ce63b5440245848383aca833047788
I20250815 01:54:09.776368 427 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 87ce63b5440245848383aca833047788
I20250815 01:54:09.776754 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 3698
I20250815 01:54:09.802213 427 minidump.cc:252] Setting minidump size limit to 20M
I20250815 01:54:09.803622 427 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:09.805028 427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:09.817579 4314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:09.819504 4317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:09.817642 4315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:09.918953 427 server_base.cc:1047] running on GCE node
I20250815 01:54:09.920323 427 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250815 01:54:09.920621 427 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250815 01:54:09.920825 427 hybrid_clock.cc:648] HybridClock initialized: now 1755222849920805 us; error 0 us; skew 500 ppm
I20250815 01:54:09.921660 427 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:09.925268 427 webserver.cc:480] Webserver started at http://0.0.0.0:37069/ using document root <none> and password file <none>
I20250815 01:54:09.926319 427 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:09.926589 427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:09.933357 427 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250815 01:54:09.937915 4322 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:09.938984 427 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250815 01:54:09.939342 427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ed6b44043527428d8b9441d35a58a3b5"
format_stamp: "Formatted at 2025-08-15 01:53:47 on dist-test-slave-h491"
I20250815 01:54:09.941663 427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:09.973487 427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:09.974920 427 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:09.975454 427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:09.984962 427 sys_catalog.cc:263] Verifying existing consensus state
W20250815 01:54:09.988833 427 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.0.106.254:45487 exists but no master address supplied!
I20250815 01:54:09.991006 427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap starting.
I20250815 01:54:10.034474 427 log.cc:826] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:10.096392 427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:10.097121 427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap complete.
I20250815 01:54:10.110845 427 raft_consensus.cc:357] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:10.111475 427 raft_consensus.cc:738] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6b44043527428d8b9441d35a58a3b5, State: Initialized, Role: FOLLOWER
I20250815 01:54:10.112143 427 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:10.112588 427 raft_consensus.cc:397] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:54:10.112798 427 raft_consensus.cc:491] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:54:10.113068 427 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 3 FOLLOWER]: Advancing to term 4
I20250815 01:54:10.118569 427 raft_consensus.cc:513] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:10.119336 427 leader_election.cc:304] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed6b44043527428d8b9441d35a58a3b5; no voters:
I20250815 01:54:10.120556 427 leader_election.cc:290] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250815 01:54:10.120808 4329 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 4 FOLLOWER]: Leader election won for term 4
I20250815 01:54:10.122081 4329 raft_consensus.cc:695] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 4 LEADER]: Becoming Leader. State: Replica: ed6b44043527428d8b9441d35a58a3b5, State: Running, Role: LEADER
I20250815 01:54:10.122861 4329 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:10.129905 4331 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6b44043527428d8b9441d35a58a3b5. Latest consensus state: current_term: 4 leader_uuid: "ed6b44043527428d8b9441d35a58a3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } } }
I20250815 01:54:10.130487 4331 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: This master's current role is: LEADER
I20250815 01:54:10.130340 4330 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "ed6b44043527428d8b9441d35a58a3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } } }
I20250815 01:54:10.131001 4330 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: This master's current role is: LEADER
I20250815 01:54:10.158478 427 tablet_replica.cc:331] stopping tablet replica
I20250815 01:54:10.159211 427 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 4 LEADER]: Raft consensus shutting down.
I20250815 01:54:10.159750 427 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250815 01:54:10.162617 427 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250815 01:54:10.163224 427 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250815 01:54:10.193045 427 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250815 01:54:10.681298 3919 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:45487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:45487: connect: Connection refused (error 111)
W20250815 01:54:10.684755 4064 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:45487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:45487: connect: Connection refused (error 111)
W20250815 01:54:10.688808 4238 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:45487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:45487: connect: Connection refused (error 111)
I20250815 01:54:15.564239 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 3768
I20250815 01:54:15.594656 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 3923
I20250815 01:54:15.623153 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 4071
I20250815 01:54:15.651094 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:45487
--webserver_interface=127.0.106.254
--webserver_port=35169
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:45487 with env {}
W20250815 01:54:15.959981 4404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:15.960600 4404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:15.961069 4404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:15.991730 4404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:54:15.992053 4404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:15.992338 4404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:54:15.992592 4404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:54:16.027776 4404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:45487
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=35169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:16.029067 4404 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:16.030705 4404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:16.041009 4410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:17.445384 4409 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 4404
W20250815 01:54:17.847714 4404 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.806s user 0.609s sys 1.196s
W20250815 01:54:16.042564 4411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:17.848222 4404 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.807s user 0.610s sys 1.196s
W20250815 01:54:17.850109 4413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:17.853459 4412 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1807 milliseconds
I20250815 01:54:17.853562 4404 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:17.854794 4404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:17.857225 4404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:17.858587 4404 hybrid_clock.cc:648] HybridClock initialized: now 1755222857858569 us; error 61 us; skew 500 ppm
I20250815 01:54:17.859365 4404 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:17.865504 4404 webserver.cc:480] Webserver started at http://127.0.106.254:35169/ using document root <none> and password file <none>
I20250815 01:54:17.866446 4404 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:17.866684 4404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:17.874420 4404 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.002s
I20250815 01:54:17.878791 4420 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:17.879798 4404 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250815 01:54:17.880110 4404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ed6b44043527428d8b9441d35a58a3b5"
format_stamp: "Formatted at 2025-08-15 01:53:47 on dist-test-slave-h491"
I20250815 01:54:17.881980 4404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:17.933521 4404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:17.935058 4404 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:17.935501 4404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:18.005123 4404 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:45487
I20250815 01:54:18.005194 4471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:45487 every 8 connection(s)
I20250815 01:54:18.007961 4404 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:54:18.008555 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 4404
I20250815 01:54:18.009995 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:33607
--local_ip_for_outbound_sockets=127.0.106.193
--tserver_master_addrs=127.0.106.254:45487
--webserver_port=39285
--webserver_interface=127.0.106.193
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:54:18.019070 4472 sys_catalog.cc:263] Verifying existing consensus state
I20250815 01:54:18.023849 4472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap starting.
I20250815 01:54:18.033722 4472 log.cc:826] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:18.109776 4472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:18.110613 4472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Bootstrap complete.
I20250815 01:54:18.129586 4472 raft_consensus.cc:357] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:18.131729 4472 raft_consensus.cc:738] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6b44043527428d8b9441d35a58a3b5, State: Initialized, Role: FOLLOWER
I20250815 01:54:18.132524 4472 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:18.133013 4472 raft_consensus.cc:397] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:54:18.133266 4472 raft_consensus.cc:491] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:54:18.133572 4472 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 5 FOLLOWER]: Advancing to term 6
I20250815 01:54:18.138821 4472 raft_consensus.cc:513] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:18.139451 4472 leader_election.cc:304] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed6b44043527428d8b9441d35a58a3b5; no voters:
I20250815 01:54:18.141587 4472 leader_election.cc:290] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250815 01:54:18.141985 4476 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 6 FOLLOWER]: Leader election won for term 6
I20250815 01:54:18.145179 4476 raft_consensus.cc:695] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [term 6 LEADER]: Becoming Leader. State: Replica: ed6b44043527428d8b9441d35a58a3b5, State: Running, Role: LEADER
I20250815 01:54:18.146080 4476 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } }
I20250815 01:54:18.146641 4472 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:54:18.155885 4477 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "ed6b44043527428d8b9441d35a58a3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } } }
I20250815 01:54:18.156589 4477 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: This master's current role is: LEADER
I20250815 01:54:18.158589 4478 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6b44043527428d8b9441d35a58a3b5. Latest consensus state: current_term: 6 leader_uuid: "ed6b44043527428d8b9441d35a58a3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6b44043527428d8b9441d35a58a3b5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 45487 } } }
I20250815 01:54:18.159276 4478 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5 [sys.catalog]: This master's current role is: LEADER
I20250815 01:54:18.169796 4483 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:54:18.183440 4483 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=01551f9940ad40d7bf41313fbf7808d8]
I20250815 01:54:18.185294 4483 catalog_manager.cc:671] Loaded metadata for table TestTable [id=608f0ed98a1d4f82b0574de82f66471d]
I20250815 01:54:18.187784 4483 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=9ee2f5eb83b34e1cbf2f82fceb9e5f5f]
I20250815 01:54:18.197860 4483 tablet_loader.cc:96] loaded metadata for tablet 2cb5bf07f99b478e8157257986a2c575 (table TestTable2 [id=01551f9940ad40d7bf41313fbf7808d8])
I20250815 01:54:18.200591 4483 tablet_loader.cc:96] loaded metadata for tablet 64f6a349284d446aad7ce94647d9ef7e (table TestTable1 [id=9ee2f5eb83b34e1cbf2f82fceb9e5f5f])
I20250815 01:54:18.202136 4483 tablet_loader.cc:96] loaded metadata for tablet 87ce63b5440245848383aca833047788 (table TestTable [id=608f0ed98a1d4f82b0574de82f66471d])
I20250815 01:54:18.203516 4483 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:54:18.209429 4483 catalog_manager.cc:1261] Loaded cluster ID: 84c21cc181284ba3849a390a775a93a0
I20250815 01:54:18.209800 4483 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:54:18.218815 4483 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:54:18.224774 4483 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ed6b44043527428d8b9441d35a58a3b5: Loaded TSK: 0
I20250815 01:54:18.226428 4483 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 01:54:18.355931 4474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:18.356508 4474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:18.357056 4474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:18.389143 4474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:18.390020 4474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:54:18.425515 4474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:33607
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=39285
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:18.426847 4474 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:18.428499 4474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:18.440927 4499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:19.846928 4498 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4474
W20250815 01:54:20.269528 4474 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.827s user 0.612s sys 1.104s
W20250815 01:54:20.270236 4474 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.828s user 0.612s sys 1.104s
W20250815 01:54:18.443077 4500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:20.271864 4502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:20.271942 4474 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 01:54:20.271198 4501 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1827 milliseconds
I20250815 01:54:20.276127 4474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:20.278244 4474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:20.279635 4474 hybrid_clock.cc:648] HybridClock initialized: now 1755222860279584 us; error 63 us; skew 500 ppm
I20250815 01:54:20.280401 4474 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:20.286623 4474 webserver.cc:480] Webserver started at http://127.0.106.193:39285/ using document root <none> and password file <none>
I20250815 01:54:20.287520 4474 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:20.287742 4474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:20.295711 4474 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20250815 01:54:20.300948 4509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:20.302130 4474 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250815 01:54:20.302500 4474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f3b8bf9a081a43049b0d43c3471b7a77"
format_stamp: "Formatted at 2025-08-15 01:53:49 on dist-test-slave-h491"
I20250815 01:54:20.304493 4474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:20.362835 4474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:20.364398 4474 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:20.364853 4474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:20.367938 4474 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:20.374290 4516 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250815 01:54:20.386112 4474 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250815 01:54:20.386376 4474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.001s
I20250815 01:54:20.386652 4474 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250815 01:54:20.392132 4516 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap starting.
I20250815 01:54:20.397176 4474 ts_tablet_manager.cc:610] Registered 2 tablets
I20250815 01:54:20.397498 4474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.008s sys 0.000s
I20250815 01:54:20.473342 4516 log.cc:826] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:20.592963 4474 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:33607
I20250815 01:54:20.593147 4623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:33607 every 8 connection(s)
I20250815 01:54:20.596462 4474 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:54:20.599092 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 4474
I20250815 01:54:20.600940 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:38517
--local_ip_for_outbound_sockets=127.0.106.194
--tserver_master_addrs=127.0.106.254:45487
--webserver_port=42319
--webserver_interface=127.0.106.194
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:54:20.625128 4516 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:20.626325 4516 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap complete.
I20250815 01:54:20.629046 4516 ts_tablet_manager.cc:1397] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Time spent bootstrapping tablet: real 0.237s user 0.197s sys 0.026s
I20250815 01:54:20.652154 4516 raft_consensus.cc:357] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:20.655387 4516 raft_consensus.cc:738] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Initialized, Role: FOLLOWER
I20250815 01:54:20.656446 4516 consensus_queue.cc:260] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:20.659792 4624 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:54:20.660320 4624 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:20.661689 4624 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:54:20.662400 4516 ts_tablet_manager.cc:1428] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77: Time spent starting tablet: real 0.033s user 0.028s sys 0.000s
I20250815 01:54:20.664757 4516 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap starting.
I20250815 01:54:20.667294 4437 ts_manager.cc:194] Registered new tserver with Master: f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193:33607)
I20250815 01:54:20.672166 4437 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: config changed from index -1 to 12, term changed from 0 to 2, VOTER ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) added, VOTER d80095fdc0a04827843c9b49656f7606 (127.0.106.195) added, VOTER f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193) added. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } } }
I20250815 01:54:20.762377 4437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:60367
I20250815 01:54:20.767400 4624 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
I20250815 01:54:20.828043 4516 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:20.828954 4516 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Bootstrap complete.
I20250815 01:54:20.830615 4516 ts_tablet_manager.cc:1397] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Time spent bootstrapping tablet: real 0.166s user 0.136s sys 0.026s
I20250815 01:54:20.832798 4516 raft_consensus.cc:357] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:20.833444 4516 raft_consensus.cc:738] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Initialized, Role: FOLLOWER
I20250815 01:54:20.834039 4516 consensus_queue.cc:260] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:20.835855 4516 ts_tablet_manager.cc:1428] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250815 01:54:20.990773 4628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:20.991328 4628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:20.991832 4628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:21.024361 4628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:21.025267 4628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:54:21.061520 4628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:38517
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=42319
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:21.063012 4628 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:21.064882 4628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:21.078188 4640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:21.916880 4646 raft_consensus.cc:491] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:54:21.917902 4646 raft_consensus.cc:513] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:21.932462 4646 leader_election.cc:290] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135), ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517)
W20250815 01:54:21.934803 4513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.194:38517: connect: Connection refused (error 111)
W20250815 01:54:21.950237 4513 leader_election.cc:336] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517): Network error: Client connection negotiation failed: client connection to 127.0.106.194:38517: connect: Connection refused (error 111)
W20250815 01:54:21.951660 4510 leader_election.cc:336] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Network error: Client connection negotiation failed: client connection to 127.0.106.195:36135: connect: Connection refused (error 111)
I20250815 01:54:21.952419 4510 leader_election.cc:304] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3b8bf9a081a43049b0d43c3471b7a77; no voters: ae4e916a450843e1995848bfe38c7e79, d80095fdc0a04827843c9b49656f7606
I20250815 01:54:21.953886 4646 raft_consensus.cc:2747] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250815 01:54:22.665558 4650 raft_consensus.cc:491] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:54:22.666365 4650 raft_consensus.cc:513] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:22.677889 4650 leader_election.cc:290] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517), d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135)
W20250815 01:54:22.682947 4510 leader_election.cc:336] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Network error: Client connection negotiation failed: client connection to 127.0.106.195:36135: connect: Connection refused (error 111)
W20250815 01:54:22.684916 4513 leader_election.cc:336] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517): Network error: Client connection negotiation failed: client connection to 127.0.106.194:38517: connect: Connection refused (error 111)
I20250815 01:54:22.685604 4513 leader_election.cc:304] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3b8bf9a081a43049b0d43c3471b7a77; no voters: ae4e916a450843e1995848bfe38c7e79, d80095fdc0a04827843c9b49656f7606
I20250815 01:54:22.686695 4650 raft_consensus.cc:2747] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250815 01:54:22.481582 4639 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4628
W20250815 01:54:22.839792 4628 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.760s user 0.000s sys 0.002s
W20250815 01:54:21.081029 4641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:22.840142 4628 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.760s user 0.000s sys 0.002s
W20250815 01:54:22.840426 4642 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1759 milliseconds
W20250815 01:54:22.842008 4643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:22.842007 4628 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:22.845690 4628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:22.847882 4628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:22.849263 4628 hybrid_clock.cc:648] HybridClock initialized: now 1755222862849232 us; error 34 us; skew 500 ppm
I20250815 01:54:22.850064 4628 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:22.857298 4628 webserver.cc:480] Webserver started at http://127.0.106.194:42319/ using document root <none> and password file <none>
I20250815 01:54:22.858234 4628 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:22.858487 4628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:22.866076 4628 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.005s
I20250815 01:54:22.871209 4658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:22.872486 4628 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250815 01:54:22.872821 4628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ae4e916a450843e1995848bfe38c7e79"
format_stamp: "Formatted at 2025-08-15 01:53:50 on dist-test-slave-h491"
I20250815 01:54:22.874938 4628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:22.927575 4628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:22.929060 4628 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:22.929504 4628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:22.932015 4628 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:22.937750 4665 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250815 01:54:22.958426 4628 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250815 01:54:22.958689 4628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.022s user 0.002s sys 0.000s
I20250815 01:54:22.959004 4628 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250815 01:54:22.964305 4665 tablet_bootstrap.cc:492] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Bootstrap starting.
I20250815 01:54:22.974684 4628 ts_tablet_manager.cc:610] Registered 3 tablets
I20250815 01:54:22.974978 4628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.016s user 0.009s sys 0.007s
I20250815 01:54:23.037230 4665 log.cc:826] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:23.111704 4665 tablet_bootstrap.cc:492] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:23.112600 4665 tablet_bootstrap.cc:492] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Bootstrap complete.
I20250815 01:54:23.114045 4665 ts_tablet_manager.cc:1397] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Time spent bootstrapping tablet: real 0.150s user 0.118s sys 0.028s
I20250815 01:54:23.131388 4665 raft_consensus.cc:357] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:54:23.134217 4665 raft_consensus.cc:738] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Initialized, Role: FOLLOWER
I20250815 01:54:23.135135 4665 consensus_queue.cc:260] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:54:23.135651 4665 raft_consensus.cc:397] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:54:23.136093 4665 raft_consensus.cc:491] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:54:23.136714 4665 raft_consensus.cc:3058] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Advancing to term 3
I20250815 01:54:23.143373 4665 raft_consensus.cc:513] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:54:23.144451 4665 leader_election.cc:304] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4e916a450843e1995848bfe38c7e79; no voters:
I20250815 01:54:23.146821 4665 leader_election.cc:290] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250815 01:54:23.147215 4762 raft_consensus.cc:2802] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 01:54:23.150651 4762 raft_consensus.cc:695] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [term 3 LEADER]: Becoming Leader. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Running, Role: LEADER
I20250815 01:54:23.151795 4762 consensus_queue.cc:237] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } }
I20250815 01:54:23.155400 4665 ts_tablet_manager.cc:1428] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79: Time spent starting tablet: real 0.041s user 0.034s sys 0.008s
I20250815 01:54:23.156103 4665 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Bootstrap starting.
I20250815 01:54:23.171264 4628 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:38517
I20250815 01:54:23.171453 4777 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:38517 every 8 connection(s)
I20250815 01:54:23.174840 4628 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 01:54:23.184216 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 4628
I20250815 01:54:23.186319 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:36135
--local_ip_for_outbound_sockets=127.0.106.195
--tserver_master_addrs=127.0.106.254:45487
--webserver_port=33931
--webserver_interface=127.0.106.195
--builtin_ntp_servers=127.0.106.212:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 01:54:23.208102 4778 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:54:23.208530 4778 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:23.209642 4778 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:54:23.213948 4436 ts_manager.cc:194] Registered new tserver with Master: ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517)
I20250815 01:54:23.215646 4436 catalog_manager.cc:5582] T 2cb5bf07f99b478e8157257986a2c575 P ae4e916a450843e1995848bfe38c7e79 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "ae4e916a450843e1995848bfe38c7e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } health_report { overall_health: HEALTHY } } }
I20250815 01:54:23.226495 4436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:57379
I20250815 01:54:23.229725 4778 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
I20250815 01:54:23.293509 4665 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:23.294415 4665 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Bootstrap complete.
I20250815 01:54:23.296090 4665 ts_tablet_manager.cc:1397] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Time spent bootstrapping tablet: real 0.140s user 0.113s sys 0.023s
I20250815 01:54:23.298209 4665 raft_consensus.cc:357] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:23.298913 4665 raft_consensus.cc:738] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Initialized, Role: FOLLOWER
I20250815 01:54:23.299543 4665 consensus_queue.cc:260] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:23.301383 4665 ts_tablet_manager.cc:1428] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250815 01:54:23.302052 4665 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Bootstrap starting.
I20250815 01:54:23.410393 4665 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:23.411077 4665 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Bootstrap complete.
I20250815 01:54:23.412284 4665 ts_tablet_manager.cc:1397] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Time spent bootstrapping tablet: real 0.110s user 0.100s sys 0.008s
I20250815 01:54:23.413780 4665 raft_consensus.cc:357] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:23.414201 4665 raft_consensus.cc:738] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e916a450843e1995848bfe38c7e79, State: Initialized, Role: FOLLOWER
I20250815 01:54:23.414779 4665 consensus_queue.cc:260] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:23.416752 4665 ts_tablet_manager.cc:1428] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250815 01:54:23.540145 4782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:23.540670 4782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:23.541154 4782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:23.572574 4782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:23.573455 4782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:54:23.608903 4782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:36135
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=33931
--tserver_master_addrs=127.0.106.254:45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:23.610252 4782 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:23.611900 4782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:23.623860 4794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:23.752400 4800 raft_consensus.cc:491] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:54:23.753415 4800 raft_consensus.cc:513] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:23.786206 4800 leader_election.cc:290] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135), ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517)
W20250815 01:54:23.816872 4510 leader_election.cc:336] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Network error: Client connection negotiation failed: client connection to 127.0.106.195:36135: connect: Connection refused (error 111)
I20250815 01:54:23.827023 4728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64f6a349284d446aad7ce94647d9ef7e" candidate_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ae4e916a450843e1995848bfe38c7e79" is_pre_election: true
I20250815 01:54:23.828116 4728 raft_consensus.cc:2466] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b8bf9a081a43049b0d43c3471b7a77 in term 2.
I20250815 01:54:23.830833 4513 leader_election.cc:304] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae4e916a450843e1995848bfe38c7e79, f3b8bf9a081a43049b0d43c3471b7a77; no voters: d80095fdc0a04827843c9b49656f7606
I20250815 01:54:23.832310 4800 raft_consensus.cc:2802] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250815 01:54:23.832742 4800 raft_consensus.cc:491] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:54:23.833665 4800 raft_consensus.cc:3058] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Advancing to term 3
I20250815 01:54:23.848205 4800 raft_consensus.cc:513] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
W20250815 01:54:23.857206 4510 leader_election.cc:336] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Network error: Client connection negotiation failed: client connection to 127.0.106.195:36135: connect: Connection refused (error 111)
I20250815 01:54:23.860049 4728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64f6a349284d446aad7ce94647d9ef7e" candidate_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ae4e916a450843e1995848bfe38c7e79"
I20250815 01:54:23.860769 4728 raft_consensus.cc:3058] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Advancing to term 3
I20250815 01:54:23.870498 4800 leader_election.cc:290] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 election: Requested vote from peers d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135), ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517)
I20250815 01:54:23.869928 4728 raft_consensus.cc:2466] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b8bf9a081a43049b0d43c3471b7a77 in term 3.
I20250815 01:54:23.872579 4513 leader_election.cc:304] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae4e916a450843e1995848bfe38c7e79, f3b8bf9a081a43049b0d43c3471b7a77; no voters: d80095fdc0a04827843c9b49656f7606
I20250815 01:54:23.874575 4800 raft_consensus.cc:2802] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 01:54:23.883522 4800 raft_consensus.cc:695] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [term 3 LEADER]: Becoming Leader. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Running, Role: LEADER
I20250815 01:54:23.885061 4800 consensus_queue.cc:237] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:23.910182 4436 catalog_manager.cc:5582] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: term changed from 2 to 3, leader changed from d80095fdc0a04827843c9b49656f7606 (127.0.106.195) to f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193). New cstate: current_term: 3 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250815 01:54:24.312181 4510 consensus_peers.cc:489] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 -> Peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Couldn't send request to peer d80095fdc0a04827843c9b49656f7606. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.195:36135: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 01:54:24.340955 4728 raft_consensus.cc:1273] T 64f6a349284d446aad7ce94647d9ef7e P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Refusing update from remote peer f3b8bf9a081a43049b0d43c3471b7a77: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250815 01:54:24.346952 4800 consensus_queue.cc:1035] T 64f6a349284d446aad7ce94647d9ef7e P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20250815 01:54:24.539798 4812 raft_consensus.cc:491] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:54:24.540799 4812 raft_consensus.cc:513] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:24.545934 4728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87ce63b5440245848383aca833047788" candidate_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ae4e916a450843e1995848bfe38c7e79" is_pre_election: true
I20250815 01:54:24.547044 4728 raft_consensus.cc:2466] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b8bf9a081a43049b0d43c3471b7a77 in term 2.
I20250815 01:54:24.549289 4513 leader_election.cc:304] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae4e916a450843e1995848bfe38c7e79, f3b8bf9a081a43049b0d43c3471b7a77; no voters:
I20250815 01:54:24.550722 4800 raft_consensus.cc:2802] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250815 01:54:24.551225 4800 raft_consensus.cc:491] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:54:24.551707 4800 raft_consensus.cc:3058] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 2 FOLLOWER]: Advancing to term 3
W20250815 01:54:24.570734 4510 leader_election.cc:336] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Network error: Client connection negotiation failed: client connection to 127.0.106.195:36135: connect: Connection refused (error 111)
I20250815 01:54:24.571882 4812 leader_election.cc:290] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517), d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135)
I20250815 01:54:24.573807 4800 raft_consensus.cc:513] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:24.578436 4728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "87ce63b5440245848383aca833047788" candidate_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ae4e916a450843e1995848bfe38c7e79"
I20250815 01:54:24.579329 4728 raft_consensus.cc:3058] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 2 FOLLOWER]: Advancing to term 3
W20250815 01:54:24.600422 4510 leader_election.cc:336] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Network error: Client connection negotiation failed: client connection to 127.0.106.195:36135: connect: Connection refused (error 111)
I20250815 01:54:24.601541 4800 leader_election.cc:290] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 election: Requested vote from peers ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517), d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135)
I20250815 01:54:24.590099 4728 raft_consensus.cc:2466] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b8bf9a081a43049b0d43c3471b7a77 in term 3.
I20250815 01:54:24.603854 4513 leader_election.cc:304] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae4e916a450843e1995848bfe38c7e79, f3b8bf9a081a43049b0d43c3471b7a77; no voters: d80095fdc0a04827843c9b49656f7606
I20250815 01:54:24.605829 4800 raft_consensus.cc:2802] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 01:54:24.607618 4800 raft_consensus.cc:695] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 3 LEADER]: Becoming Leader. State: Replica: f3b8bf9a081a43049b0d43c3471b7a77, State: Running, Role: LEADER
I20250815 01:54:24.609198 4800 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:24.629317 4436 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: term changed from 2 to 3, leader changed from <none> to f3b8bf9a081a43049b0d43c3471b7a77 (127.0.106.193). New cstate: current_term: 3 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250815 01:54:25.094765 4510 consensus_peers.cc:489] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 -> Peer d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): Couldn't send request to peer d80095fdc0a04827843c9b49656f7606. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.195:36135: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 01:54:25.115581 4728 raft_consensus.cc:1273] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Refusing update from remote peer f3b8bf9a081a43049b0d43c3471b7a77: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250815 01:54:25.118269 4800 consensus_queue.cc:1035] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250815 01:54:25.232753 4579 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:25.246505 4728 raft_consensus.cc:1273] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Refusing update from remote peer f3b8bf9a081a43049b0d43c3471b7a77: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250815 01:54:25.248643 4822 consensus_queue.cc:1035] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20250815 01:54:25.026312 4793 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4782
W20250815 01:54:25.257851 4793 kernel_stack_watchdog.cc:198] Thread 4782 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:54:23.624491 4795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:25.257110 4800 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER d80095fdc0a04827843c9b49656f7606 (127.0.106.195) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } }
W20250815 01:54:25.263962 4782 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.640s user 0.565s sys 0.941s
W20250815 01:54:25.264358 4782 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.640s user 0.566s sys 0.942s
W20250815 01:54:25.265132 4796 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1638 milliseconds
W20250815 01:54:25.265556 4797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:25.265550 4782 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:25.265436 4728 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER d80095fdc0a04827843c9b49656f7606 (127.0.106.195) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } }
I20250815 01:54:25.267252 4782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:25.270411 4421 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 87ce63b5440245848383aca833047788 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 01:54:25.272130 4782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:25.273675 4782 hybrid_clock.cc:648] HybridClock initialized: now 1755222865273594 us; error 76 us; skew 500 ppm
I20250815 01:54:25.273440 4437 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: config changed from index 12 to 14, VOTER d80095fdc0a04827843c9b49656f7606 (127.0.106.195) evicted. New cstate: current_term: 3 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 01:54:25.274864 4782 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:25.282507 4782 webserver.cc:480] Webserver started at http://127.0.106.195:33931/ using document root <none> and password file <none>
I20250815 01:54:25.283782 4782 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:25.284027 4782 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250815 01:54:25.284597 4437 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 87ce63b5440245848383aca833047788 on TS d80095fdc0a04827843c9b49656f7606: Not found: failed to reset TS proxy: Could not find TS for UUID d80095fdc0a04827843c9b49656f7606
I20250815 01:54:25.288071 4579 consensus_queue.cc:237] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } }
I20250815 01:54:25.290096 4800 raft_consensus.cc:2953] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } }
I20250815 01:54:25.293653 4782 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.005s
I20250815 01:54:25.296818 4421 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 87ce63b5440245848383aca833047788 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 01:54:25.299649 4834 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:25.299968 4437 catalog_manager.cc:5582] T 87ce63b5440245848383aca833047788 P f3b8bf9a081a43049b0d43c3471b7a77 reported cstate change: config changed from index 14 to 15, VOTER ae4e916a450843e1995848bfe38c7e79 (127.0.106.194) evicted. New cstate: current_term: 3 leader_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } health_report { overall_health: HEALTHY } } }
I20250815 01:54:25.301046 4782 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20250815 01:54:25.301379 4782 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d80095fdc0a04827843c9b49656f7606"
format_stamp: "Formatted at 2025-08-15 01:53:52 on dist-test-slave-h491"
I20250815 01:54:25.303386 4782 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:25.318769 4708 tablet_service.cc:1515] Processing DeleteTablet for tablet 87ce63b5440245848383aca833047788 with delete_type TABLET_DATA_TOMBSTONED (TS ae4e916a450843e1995848bfe38c7e79 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:57620
I20250815 01:54:25.321461 4836 tablet_replica.cc:331] stopping tablet replica
I20250815 01:54:25.322126 4836 raft_consensus.cc:2241] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250815 01:54:25.322669 4836 raft_consensus.cc:2270] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250815 01:54:25.325332 4836 ts_tablet_manager.cc:1905] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 01:54:25.335531 4836 ts_tablet_manager.cc:1918] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250815 01:54:25.335870 4836 log.cc:1199] T 87ce63b5440245848383aca833047788 P ae4e916a450843e1995848bfe38c7e79: Deleting WAL directory at /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/wals/87ce63b5440245848383aca833047788
W20250815 01:54:25.337266 4422 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 87ce63b5440245848383aca833047788 on TS d80095fdc0a04827843c9b49656f7606 failed: Not found: failed to reset TS proxy: Could not find TS for UUID d80095fdc0a04827843c9b49656f7606
I20250815 01:54:25.337303 4424 catalog_manager.cc:4928] TS ae4e916a450843e1995848bfe38c7e79 (127.0.106.194:38517): tablet 87ce63b5440245848383aca833047788 (table TestTable [id=608f0ed98a1d4f82b0574de82f66471d]) successfully deleted
I20250815 01:54:25.384860 4782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:25.386312 4782 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:25.386785 4782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:25.389268 4782 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:25.395177 4843 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250815 01:54:25.406862 4782 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250815 01:54:25.407156 4782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.002s
I20250815 01:54:25.407426 4782 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250815 01:54:25.413311 4843 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Bootstrap starting.
I20250815 01:54:25.416127 4782 ts_tablet_manager.cc:610] Registered 2 tablets
I20250815 01:54:25.416350 4782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.007s sys 0.000s
I20250815 01:54:25.475836 4843 log.cc:826] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:25.576366 4843 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:25.577256 4843 tablet_bootstrap.cc:492] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Bootstrap complete.
I20250815 01:54:25.579135 4843 ts_tablet_manager.cc:1397] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Time spent bootstrapping tablet: real 0.166s user 0.131s sys 0.032s
I20250815 01:54:25.586367 4782 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:36135
I20250815 01:54:25.586578 4950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:36135 every 8 connection(s)
I20250815 01:54:25.588894 4782 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 01:54:25.599180 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 4782
I20250815 01:54:25.599905 4843 raft_consensus.cc:357] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:25.603678 4843 raft_consensus.cc:738] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80095fdc0a04827843c9b49656f7606, State: Initialized, Role: FOLLOWER
I20250815 01:54:25.604866 4843 consensus_queue.cc:260] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } } peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } attrs { promote: false } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } }
I20250815 01:54:25.615654 4843 ts_tablet_manager.cc:1428] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606: Time spent starting tablet: real 0.036s user 0.029s sys 0.005s
I20250815 01:54:25.616948 4843 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Bootstrap starting.
I20250815 01:54:25.640964 4951 heartbeater.cc:344] Connected to a master server at 127.0.106.254:45487
I20250815 01:54:25.641606 4951 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:25.643276 4951 heartbeater.cc:507] Master 127.0.106.254:45487 requested a full tablet report, sending...
I20250815 01:54:25.649039 4436 ts_manager.cc:194] Registered new tserver with Master: d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135)
I20250815 01:54:25.656739 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:54:25.656517 4436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:56275
I20250815 01:54:25.665177 4951 heartbeater.cc:499] Master 127.0.106.254:45487 was elected leader, sending a full tablet report...
I20250815 01:54:25.666994 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250815 01:54:25.673063 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250815 01:54:25.676685 4886 tablet_service.cc:1515] Processing DeleteTablet for tablet 87ce63b5440245848383aca833047788 with delete_type TABLET_DATA_TOMBSTONED (TS d80095fdc0a04827843c9b49656f7606 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:60650
W20250815 01:54:25.683112 4421 catalog_manager.cc:4908] TS d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): delete failed for tablet 87ce63b5440245848383aca833047788 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87ce63b5440245848383aca833047788 already in progress: opening tablet
I20250815 01:54:25.738991 4843 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:54:25.739987 4843 tablet_bootstrap.cc:492] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Bootstrap complete.
I20250815 01:54:25.741371 4843 ts_tablet_manager.cc:1397] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Time spent bootstrapping tablet: real 0.125s user 0.100s sys 0.021s
I20250815 01:54:25.743120 4843 raft_consensus.cc:357] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:25.743593 4843 raft_consensus.cc:738] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80095fdc0a04827843c9b49656f7606, State: Initialized, Role: FOLLOWER
I20250815 01:54:25.744138 4843 consensus_queue.cc:260] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f3b8bf9a081a43049b0d43c3471b7a77" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 33607 } } peers { permanent_uuid: "ae4e916a450843e1995848bfe38c7e79" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 38517 } attrs { promote: false } } peers { permanent_uuid: "d80095fdc0a04827843c9b49656f7606" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 36135 } attrs { promote: false } }
I20250815 01:54:25.745908 4843 ts_tablet_manager.cc:1428] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250815 01:54:25.750664 4886 tablet_service.cc:1515] Processing DeleteTablet for tablet 87ce63b5440245848383aca833047788 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 12 (current committed config index is 15)) from {username='slave'} at 127.0.0.1:60650
I20250815 01:54:25.751731 4960 tablet_replica.cc:331] stopping tablet replica
I20250815 01:54:25.752599 4960 raft_consensus.cc:2241] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 01:54:25.753032 4960 raft_consensus.cc:2270] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 01:54:25.755677 4960 ts_tablet_manager.cc:1905] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 01:54:25.770078 4960 ts_tablet_manager.cc:1918] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250815 01:54:25.770504 4960 log.cc:1199] T 87ce63b5440245848383aca833047788 P d80095fdc0a04827843c9b49656f7606: Deleting WAL directory at /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/wals/87ce63b5440245848383aca833047788
I20250815 01:54:25.771234 4906 raft_consensus.cc:3058] T 64f6a349284d446aad7ce94647d9ef7e P d80095fdc0a04827843c9b49656f7606 [term 2 FOLLOWER]: Advancing to term 3
I20250815 01:54:25.771888 4421 catalog_manager.cc:4928] TS d80095fdc0a04827843c9b49656f7606 (127.0.106.195:36135): tablet 87ce63b5440245848383aca833047788 (table TestTable [id=608f0ed98a1d4f82b0574de82f66471d]) successfully deleted
W20250815 01:54:26.677714 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:27.681272 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:28.684794 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:29.688316 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:30.691713 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:31.695319 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:32.698923 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:33.702708 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:34.705950 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:35.710157 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:36.714097 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:37.717548 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:38.720737 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:39.723874 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:40.727013 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:41.730319 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:42.733633 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:43.736975 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:44.740311 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 87ce63b5440245848383aca833047788: tablet_id: "87ce63b5440245848383aca833047788" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 01:54:45.278367 4947 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce1e0 after lost signal to thread 4789
W20250815 01:54:45.279184 4947 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd640 after lost signal to thread 4950
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250815 01:54:45.744562 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 4474
I20250815 01:54:45.784998 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 4628
I20250815 01:54:45.823474 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 4782
I20250815 01:54:45.853377 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 4404
2025-08-15T01:54:45Z chronyd exiting
I20250815 01:54:45.902393 427 test_util.cc:183] -----------------------------------------------
I20250815 01:54:45.902607 427 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755222754379616-427-0
[ FAILED ] AdminCliTest.TestRebuildTables (61073 ms)
[----------] 5 tests from AdminCliTest (131456 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250815 01:54:45.906458 427 test_util.cc:276] Using random seed: -1398949886
I20250815 01:54:45.911054 427 ts_itest-base.cc:115] Starting cluster with:
I20250815 01:54:45.911211 427 ts_itest-base.cc:116] --------------
I20250815 01:54:45.911360 427 ts_itest-base.cc:117] 5 tablet servers
I20250815 01:54:45.911504 427 ts_itest-base.cc:118] 3 replicas per TS
I20250815 01:54:45.911634 427 ts_itest-base.cc:119] --------------
2025-08-15T01:54:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T01:54:45Z Disabled control of system clock
I20250815 01:54:45.954449 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:32789
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:32789
--raft_prepare_replacement_before_eviction=true with env {}
W20250815 01:54:46.269902 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:46.270486 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:46.270889 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:46.302706 4985 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 01:54:46.303056 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:54:46.303268 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:46.303472 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:54:46.303668 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:54:46.338811 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:32789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:46.340112 4985 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:46.341634 4985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:46.351959 4991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:47.756431 4990 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 4985
W20250815 01:54:48.168908 4985 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.817s user 0.643s sys 1.173s
W20250815 01:54:46.352615 4992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:48.169423 4985 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.817s user 0.643s sys 1.173s
W20250815 01:54:48.171478 4994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:48.174187 4993 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1817 milliseconds
I20250815 01:54:48.174224 4985 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:48.175601 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:48.178138 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:48.179564 4985 hybrid_clock.cc:648] HybridClock initialized: now 1755222888179527 us; error 86 us; skew 500 ppm
I20250815 01:54:48.180387 4985 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:48.187110 4985 webserver.cc:480] Webserver started at http://127.0.106.254:34547/ using document root <none> and password file <none>
I20250815 01:54:48.188073 4985 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:48.188290 4985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:48.188766 4985 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:54:48.193554 4985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "44b65ac25d094cba81ecdf5156e53575"
format_stamp: "Formatted at 2025-08-15 01:54:48 on dist-test-slave-h491"
I20250815 01:54:48.194790 4985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "44b65ac25d094cba81ecdf5156e53575"
format_stamp: "Formatted at 2025-08-15 01:54:48 on dist-test-slave-h491"
I20250815 01:54:48.202626 4985 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250815 01:54:48.208238 5001 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:48.209259 4985 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250815 01:54:48.209609 4985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "44b65ac25d094cba81ecdf5156e53575"
format_stamp: "Formatted at 2025-08-15 01:54:48 on dist-test-slave-h491"
I20250815 01:54:48.209940 4985 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:48.267577 4985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:48.269037 4985 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:48.269475 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:48.341847 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:32789
I20250815 01:54:48.341921 5052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:32789 every 8 connection(s)
I20250815 01:54:48.344514 4985 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:54:48.349622 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 4985
I20250815 01:54:48.350162 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 01:54:48.350297 5053 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:54:48.375020 5053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575: Bootstrap starting.
I20250815 01:54:48.380829 5053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575: Neither blocks nor log segments found. Creating new log.
I20250815 01:54:48.382539 5053 log.cc:826] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:48.387055 5053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575: No bootstrap required, opened a new log
I20250815 01:54:48.405041 5053 raft_consensus.cc:357] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b65ac25d094cba81ecdf5156e53575" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 32789 } }
I20250815 01:54:48.405772 5053 raft_consensus.cc:383] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:54:48.405975 5053 raft_consensus.cc:738] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b65ac25d094cba81ecdf5156e53575, State: Initialized, Role: FOLLOWER
I20250815 01:54:48.406630 5053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b65ac25d094cba81ecdf5156e53575" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 32789 } }
I20250815 01:54:48.407121 5053 raft_consensus.cc:397] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:54:48.407349 5053 raft_consensus.cc:491] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:54:48.407636 5053 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:54:48.412133 5053 raft_consensus.cc:513] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b65ac25d094cba81ecdf5156e53575" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 32789 } }
I20250815 01:54:48.412910 5053 leader_election.cc:304] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44b65ac25d094cba81ecdf5156e53575; no voters:
I20250815 01:54:48.414786 5053 leader_election.cc:290] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:54:48.415525 5058 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:54:48.417686 5058 raft_consensus.cc:695] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [term 1 LEADER]: Becoming Leader. State: Replica: 44b65ac25d094cba81ecdf5156e53575, State: Running, Role: LEADER
I20250815 01:54:48.418479 5058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b65ac25d094cba81ecdf5156e53575" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 32789 } }
I20250815 01:54:48.419471 5053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:54:48.429091 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44b65ac25d094cba81ecdf5156e53575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b65ac25d094cba81ecdf5156e53575" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 32789 } } }
I20250815 01:54:48.429860 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44b65ac25d094cba81ecdf5156e53575. Latest consensus state: current_term: 1 leader_uuid: "44b65ac25d094cba81ecdf5156e53575" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b65ac25d094cba81ecdf5156e53575" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 32789 } } }
I20250815 01:54:48.430089 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [sys.catalog]: This master's current role is: LEADER
I20250815 01:54:48.430405 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575 [sys.catalog]: This master's current role is: LEADER
I20250815 01:54:48.433879 5066 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:54:48.445422 5066 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:54:48.460081 5066 catalog_manager.cc:1349] Generated new cluster ID: d8b4027e470b4399aa4a516ca47da8f2
I20250815 01:54:48.460417 5066 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:54:48.477926 5066 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:54:48.479596 5066 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:54:48.493964 5066 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 44b65ac25d094cba81ecdf5156e53575: Generated new TSK 0
I20250815 01:54:48.495007 5066 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 01:54:48.515702 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--builtin_ntp_servers=127.0.106.212:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250815 01:54:48.830802 5077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:48.831384 5077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:48.831889 5077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:48.863744 5077 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 01:54:48.864163 5077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:48.864961 5077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:54:48.901510 5077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:48.902943 5077 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:48.904474 5077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:48.916944 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:50.320782 5082 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 5077
W20250815 01:54:50.779232 5082 kernel_stack_watchdog.cc:198] Thread 5077 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:54:48.918301 5084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:50.780490 5085 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1861 milliseconds
W20250815 01:54:50.780757 5077 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.862s user 0.003s sys 0.000s
W20250815 01:54:50.781116 5077 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.863s user 0.003s sys 0.000s
W20250815 01:54:50.786087 5086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:50.786078 5077 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:50.787683 5077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:50.790295 5077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:50.791816 5077 hybrid_clock.cc:648] HybridClock initialized: now 1755222890791780 us; error 34 us; skew 500 ppm
I20250815 01:54:50.792883 5077 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:50.800478 5077 webserver.cc:480] Webserver started at http://127.0.106.193:43113/ using document root <none> and password file <none>
I20250815 01:54:50.801753 5077 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:50.802032 5077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:50.802644 5077 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:54:50.809132 5077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e752c8b174fa4c9ba4991e5a3fa9416c"
format_stamp: "Formatted at 2025-08-15 01:54:50 on dist-test-slave-h491"
I20250815 01:54:50.810698 5077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e752c8b174fa4c9ba4991e5a3fa9416c"
format_stamp: "Formatted at 2025-08-15 01:54:50 on dist-test-slave-h491"
I20250815 01:54:50.820204 5077 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20250815 01:54:50.827909 5093 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:50.829106 5077 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 01:54:50.829499 5077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e752c8b174fa4c9ba4991e5a3fa9416c"
format_stamp: "Formatted at 2025-08-15 01:54:50 on dist-test-slave-h491"
I20250815 01:54:50.829939 5077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:50.902580 5077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:50.904594 5077 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:50.905176 5077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:50.908491 5077 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:50.914250 5077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:54:50.914606 5077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250815 01:54:50.914935 5077 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:54:50.915099 5077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:51.086819 5077 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:43011
I20250815 01:54:51.086934 5205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:43011 every 8 connection(s)
I20250815 01:54:51.089641 5077 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:54:51.098865 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 5077
I20250815 01:54:51.099345 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 01:54:51.106995 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--builtin_ntp_servers=127.0.106.212:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 01:54:51.117937 5206 heartbeater.cc:344] Connected to a master server at 127.0.106.254:32789
I20250815 01:54:51.118438 5206 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:51.119720 5206 heartbeater.cc:507] Master 127.0.106.254:32789 requested a full tablet report, sending...
I20250815 01:54:51.122227 5018 ts_manager.cc:194] Registered new tserver with Master: e752c8b174fa4c9ba4991e5a3fa9416c (127.0.106.193:43011)
I20250815 01:54:51.124258 5018 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:57913
W20250815 01:54:51.462730 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:51.463248 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:51.463745 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:51.494733 5210 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 01:54:51.495185 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:51.496026 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:54:51.531165 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:51.532464 5210 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:51.534013 5210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:51.545513 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:52.127810 5206 heartbeater.cc:499] Master 127.0.106.254:32789 was elected leader, sending a full tablet report...
W20250815 01:54:51.547366 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:52.765883 5218 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1213 milliseconds
W20250815 01:54:52.766673 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:52.768723 5210 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.222s user 0.004s sys 0.004s
W20250815 01:54:52.769008 5210 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.223s user 0.004s sys 0.004s
I20250815 01:54:52.769217 5210 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:52.770241 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:52.772495 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:52.773836 5210 hybrid_clock.cc:648] HybridClock initialized: now 1755222892773813 us; error 44 us; skew 500 ppm
I20250815 01:54:52.774652 5210 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:52.782219 5210 webserver.cc:480] Webserver started at http://127.0.106.194:42305/ using document root <none> and password file <none>
I20250815 01:54:52.783313 5210 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:52.783537 5210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:52.783970 5210 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:54:52.788439 5210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6"
format_stamp: "Formatted at 2025-08-15 01:54:52 on dist-test-slave-h491"
I20250815 01:54:52.789572 5210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6"
format_stamp: "Formatted at 2025-08-15 01:54:52 on dist-test-slave-h491"
I20250815 01:54:52.797801 5210 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250815 01:54:52.804100 5226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:52.805351 5210 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250815 01:54:52.805672 5210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6"
format_stamp: "Formatted at 2025-08-15 01:54:52 on dist-test-slave-h491"
I20250815 01:54:52.806010 5210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:52.889878 5210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:52.891413 5210 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:52.891847 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:52.894330 5210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:52.898437 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:54:52.898649 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:52.898883 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:54:52.899041 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:53.032316 5210 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:45325
I20250815 01:54:53.032435 5338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:45325 every 8 connection(s)
I20250815 01:54:53.034839 5210 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 01:54:53.037824 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 5210
I20250815 01:54:53.038544 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 01:54:53.044735 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--builtin_ntp_servers=127.0.106.212:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 01:54:53.056413 5339 heartbeater.cc:344] Connected to a master server at 127.0.106.254:32789
I20250815 01:54:53.056955 5339 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:53.058286 5339 heartbeater.cc:507] Master 127.0.106.254:32789 requested a full tablet report, sending...
I20250815 01:54:53.060694 5018 ts_manager.cc:194] Registered new tserver with Master: 55cc1d7c8ccc40eaae8f854d4095e4b6 (127.0.106.194:45325)
I20250815 01:54:53.061895 5018 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:59595
W20250815 01:54:53.360288 5343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:53.360795 5343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:53.361282 5343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:53.392885 5343 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 01:54:53.393332 5343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:53.394114 5343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:54:53.429966 5343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:53.431553 5343 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:53.433110 5343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:53.444926 5349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:54.065568 5339 heartbeater.cc:499] Master 127.0.106.254:32789 was elected leader, sending a full tablet report...
W20250815 01:54:53.445578 5350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:54.736052 5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:54.738495 5343 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.294s user 0.493s sys 0.794s
W20250815 01:54:54.738781 5343 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.294s user 0.493s sys 0.794s
W20250815 01:54:54.739557 5351 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1288 milliseconds
I20250815 01:54:54.739624 5343 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:54.740808 5343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:54.746515 5343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:54.747853 5343 hybrid_clock.cc:648] HybridClock initialized: now 1755222894747831 us; error 47 us; skew 500 ppm
I20250815 01:54:54.748634 5343 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:54.755959 5343 webserver.cc:480] Webserver started at http://127.0.106.195:42593/ using document root <none> and password file <none>
I20250815 01:54:54.756870 5343 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:54.757092 5343 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:54.757535 5343 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:54:54.763070 5343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "23bda103fa024f0d86d95a8465515c27"
format_stamp: "Formatted at 2025-08-15 01:54:54 on dist-test-slave-h491"
I20250815 01:54:54.764148 5343 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "23bda103fa024f0d86d95a8465515c27"
format_stamp: "Formatted at 2025-08-15 01:54:54 on dist-test-slave-h491"
I20250815 01:54:54.772933 5343 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.000s sys 0.009s
I20250815 01:54:54.778944 5360 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:54.780098 5343 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250815 01:54:54.780407 5343 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "23bda103fa024f0d86d95a8465515c27"
format_stamp: "Formatted at 2025-08-15 01:54:54 on dist-test-slave-h491"
I20250815 01:54:54.780740 5343 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:54.842175 5343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:54.843621 5343 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:54.844035 5343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:54.846565 5343 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:54.850615 5343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:54:54.850827 5343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:54.851063 5343 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:54:54.851218 5343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:54.983891 5343 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:43819
I20250815 01:54:54.984001 5472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:43819 every 8 connection(s)
I20250815 01:54:54.986562 5343 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 01:54:54.991765 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 5343
I20250815 01:54:54.992362 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 01:54:55.003185 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.196:0
--local_ip_for_outbound_sockets=127.0.106.196
--webserver_interface=127.0.106.196
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--builtin_ntp_servers=127.0.106.212:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 01:54:55.011339 5473 heartbeater.cc:344] Connected to a master server at 127.0.106.254:32789
I20250815 01:54:55.011894 5473 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:55.013248 5473 heartbeater.cc:507] Master 127.0.106.254:32789 requested a full tablet report, sending...
I20250815 01:54:55.015797 5018 ts_manager.cc:194] Registered new tserver with Master: 23bda103fa024f0d86d95a8465515c27 (127.0.106.195:43819)
I20250815 01:54:55.017612 5018 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:60271
W20250815 01:54:55.316470 5477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:55.316938 5477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:55.317425 5477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:55.349495 5477 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 01:54:55.349908 5477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:55.350747 5477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.196
I20250815 01:54:55.385699 5477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.106.196
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.196
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:55.387022 5477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:55.388597 5477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:55.401391 5483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:56.020998 5473 heartbeater.cc:499] Master 127.0.106.254:32789 was elected leader, sending a full tablet report...
W20250815 01:54:55.401578 5484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:56.749516 5485 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1342 milliseconds
W20250815 01:54:56.751077 5486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:56.753516 5477 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.352s user 0.000s sys 0.009s
W20250815 01:54:56.753886 5477 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.353s user 0.000s sys 0.009s
I20250815 01:54:56.754174 5477 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:56.755887 5477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:56.758783 5477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:56.760303 5477 hybrid_clock.cc:648] HybridClock initialized: now 1755222896760244 us; error 54 us; skew 500 ppm
I20250815 01:54:56.761473 5477 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:56.770460 5477 webserver.cc:480] Webserver started at http://127.0.106.196:40359/ using document root <none> and password file <none>
I20250815 01:54:56.771927 5477 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:56.772253 5477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:56.772923 5477 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:54:56.780371 5477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "adb0d6141d0d40b199033b7441c3f659"
format_stamp: "Formatted at 2025-08-15 01:54:56 on dist-test-slave-h491"
I20250815 01:54:56.782038 5477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "adb0d6141d0d40b199033b7441c3f659"
format_stamp: "Formatted at 2025-08-15 01:54:56 on dist-test-slave-h491"
I20250815 01:54:56.792480 5477 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.005s sys 0.003s
I20250815 01:54:56.800869 5494 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:56.802162 5477 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250815 01:54:56.802611 5477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "adb0d6141d0d40b199033b7441c3f659"
format_stamp: "Formatted at 2025-08-15 01:54:56 on dist-test-slave-h491"
I20250815 01:54:56.803084 5477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:56.911695 5477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:56.913220 5477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:56.913635 5477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:56.916180 5477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:56.920434 5477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:54:56.920653 5477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:56.920867 5477 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:54:56.921005 5477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:57.065742 5477 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.196:34343
I20250815 01:54:57.065863 5606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.196:34343 every 8 connection(s)
I20250815 01:54:57.068393 5477 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250815 01:54:57.075146 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 5477
I20250815 01:54:57.075647 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250815 01:54:57.082186 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.197:0
--local_ip_for_outbound_sockets=127.0.106.197
--webserver_interface=127.0.106.197
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--builtin_ntp_servers=127.0.106.212:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 01:54:57.093886 5607 heartbeater.cc:344] Connected to a master server at 127.0.106.254:32789
I20250815 01:54:57.094313 5607 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:57.095371 5607 heartbeater.cc:507] Master 127.0.106.254:32789 requested a full tablet report, sending...
I20250815 01:54:57.097455 5018 ts_manager.cc:194] Registered new tserver with Master: adb0d6141d0d40b199033b7441c3f659 (127.0.106.196:34343)
I20250815 01:54:57.099102 5018 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.196:52995
W20250815 01:54:57.392496 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:54:57.392969 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:54:57.393406 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:54:57.425747 5611 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 01:54:57.426090 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:54:57.426888 5611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.197
I20250815 01:54:57.462666 5611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.106.197
--webserver_port=0
--tserver_master_addrs=127.0.106.254:32789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.197
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:54:57.463877 5611 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:54:57.465396 5611 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:54:57.477325 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:54:58.102274 5607 heartbeater.cc:499] Master 127.0.106.254:32789 was elected leader, sending a full tablet report...
W20250815 01:54:57.478416 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:58.697242 5620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:54:58.699589 5619 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1217 milliseconds
I20250815 01:54:58.699676 5611 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:54:58.700886 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:54:58.704234 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:54:58.705664 5611 hybrid_clock.cc:648] HybridClock initialized: now 1755222898705617 us; error 60 us; skew 500 ppm
I20250815 01:54:58.706497 5611 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:54:58.717730 5611 webserver.cc:480] Webserver started at http://127.0.106.197:45449/ using document root <none> and password file <none>
I20250815 01:54:58.718973 5611 fs_manager.cc:362] Metadata directory not provided
I20250815 01:54:58.719264 5611 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:54:58.719735 5611 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:54:58.724186 5611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "cb0a89252338428db395418a4e1d38af"
format_stamp: "Formatted at 2025-08-15 01:54:58 on dist-test-slave-h491"
I20250815 01:54:58.725255 5611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "cb0a89252338428db395418a4e1d38af"
format_stamp: "Formatted at 2025-08-15 01:54:58 on dist-test-slave-h491"
I20250815 01:54:58.732647 5611 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250815 01:54:58.738960 5627 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:58.740021 5611 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 01:54:58.740335 5611 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cb0a89252338428db395418a4e1d38af"
format_stamp: "Formatted at 2025-08-15 01:54:58 on dist-test-slave-h491"
I20250815 01:54:58.740664 5611 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:54:58.813387 5611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:54:58.814865 5611 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:54:58.815294 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:54:58.818540 5611 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:54:58.824725 5611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:54:58.825035 5611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:58.825354 5611 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:54:58.825582 5611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:54:58.983371 5611 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.197:36233
I20250815 01:54:58.983947 5740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.197:36233 every 8 connection(s)
I20250815 01:54:58.987095 5611 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250815 01:54:58.992733 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 5611
I20250815 01:54:58.993256 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755222754379616-427-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250815 01:54:59.020790 5741 heartbeater.cc:344] Connected to a master server at 127.0.106.254:32789
I20250815 01:54:59.021229 5741 heartbeater.cc:461] Registering TS with master...
I20250815 01:54:59.022245 5741 heartbeater.cc:507] Master 127.0.106.254:32789 requested a full tablet report, sending...
I20250815 01:54:59.024379 5018 ts_manager.cc:194] Registered new tserver with Master: cb0a89252338428db395418a4e1d38af (127.0.106.197:36233)
I20250815 01:54:59.026062 5018 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.197:60287
I20250815 01:54:59.032889 427 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250815 01:54:59.073480 5018 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51594:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 01:54:59.163636 5141 tablet_service.cc:1468] Processing CreateTablet for tablet 13e34a99ab9646d6ab15e3de87215077 (DEFAULT_TABLE table=TestTable [id=4db28bb74ac04829a00fe72492973d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:54:59.166311 5141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e34a99ab9646d6ab15e3de87215077. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:54:59.171382 5274 tablet_service.cc:1468] Processing CreateTablet for tablet 13e34a99ab9646d6ab15e3de87215077 (DEFAULT_TABLE table=TestTable [id=4db28bb74ac04829a00fe72492973d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:54:59.171655 5542 tablet_service.cc:1468] Processing CreateTablet for tablet 13e34a99ab9646d6ab15e3de87215077 (DEFAULT_TABLE table=TestTable [id=4db28bb74ac04829a00fe72492973d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:54:59.173070 5274 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e34a99ab9646d6ab15e3de87215077. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:54:59.174094 5542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13e34a99ab9646d6ab15e3de87215077. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:54:59.196854 5760 tablet_bootstrap.cc:492] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6: Bootstrap starting.
I20250815 01:54:59.200736 5761 tablet_bootstrap.cc:492] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c: Bootstrap starting.
I20250815 01:54:59.204644 5762 tablet_bootstrap.cc:492] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659: Bootstrap starting.
I20250815 01:54:59.205991 5760 tablet_bootstrap.cc:654] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6: Neither blocks nor log segments found. Creating new log.
I20250815 01:54:59.208830 5760 log.cc:826] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:59.210140 5761 tablet_bootstrap.cc:654] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c: Neither blocks nor log segments found. Creating new log.
I20250815 01:54:59.212097 5762 tablet_bootstrap.cc:654] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659: Neither blocks nor log segments found. Creating new log.
I20250815 01:54:59.212612 5761 log.cc:826] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:59.214675 5762 log.cc:826] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659: Log is configured to *not* fsync() on all Append() calls
I20250815 01:54:59.215209 5760 tablet_bootstrap.cc:492] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6: No bootstrap required, opened a new log
I20250815 01:54:59.215664 5760 ts_tablet_manager.cc:1397] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.012s
I20250815 01:54:59.218791 5761 tablet_bootstrap.cc:492] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c: No bootstrap required, opened a new log
I20250815 01:54:59.219326 5761 ts_tablet_manager.cc:1397] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20250815 01:54:59.221158 5762 tablet_bootstrap.cc:492] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659: No bootstrap required, opened a new log
I20250815 01:54:59.221663 5762 ts_tablet_manager.cc:1397] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.008s
I20250815 01:54:59.237380 5760 raft_consensus.cc:357] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
I20250815 01:54:59.238123 5760 raft_consensus.cc:383] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:54:59.238406 5760 raft_consensus.cc:738] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55cc1d7c8ccc40eaae8f854d4095e4b6, State: Initialized, Role: FOLLOWER
I20250815 01:54:59.239187 5760 consensus_queue.cc:260] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
I20250815 01:54:59.243808 5760 ts_tablet_manager.cc:1428] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6: Time spent starting tablet: real 0.028s user 0.018s sys 0.009s
I20250815 01:54:59.248463 5761 raft_consensus.cc:357] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
I20250815 01:54:59.249171 5761 raft_consensus.cc:383] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:54:59.249488 5761 raft_consensus.cc:738] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e752c8b174fa4c9ba4991e5a3fa9416c, State: Initialized, Role: FOLLOWER
I20250815 01:54:59.250440 5761 consensus_queue.cc:260] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
I20250815 01:54:59.250960 5762 raft_consensus.cc:357] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
I20250815 01:54:59.252480 5762 raft_consensus.cc:383] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:54:59.252807 5762 raft_consensus.cc:738] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb0d6141d0d40b199033b7441c3f659, State: Initialized, Role: FOLLOWER
I20250815 01:54:59.253784 5762 consensus_queue.cc:260] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
I20250815 01:54:59.255345 5761 ts_tablet_manager.cc:1428] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c: Time spent starting tablet: real 0.036s user 0.025s sys 0.010s
I20250815 01:54:59.259554 5762 ts_tablet_manager.cc:1428] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659: Time spent starting tablet: real 0.038s user 0.032s sys 0.005s
W20250815 01:54:59.296069 5340 tablet.cc:2378] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:54:59.305619 5766 raft_consensus.cc:491] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:54:59.306149 5766 raft_consensus.cc:513] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
I20250815 01:54:59.308607 5766 leader_election.cc:290] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e752c8b174fa4c9ba4991e5a3fa9416c (127.0.106.193:43011), adb0d6141d0d40b199033b7441c3f659 (127.0.106.196:34343)
I20250815 01:54:59.320968 5161 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e34a99ab9646d6ab15e3de87215077" candidate_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" is_pre_election: true
I20250815 01:54:59.322279 5161 raft_consensus.cc:2466] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55cc1d7c8ccc40eaae8f854d4095e4b6 in term 0.
I20250815 01:54:59.323570 5230 leader_election.cc:304] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55cc1d7c8ccc40eaae8f854d4095e4b6, e752c8b174fa4c9ba4991e5a3fa9416c; no voters:
I20250815 01:54:59.323741 5562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e34a99ab9646d6ab15e3de87215077" candidate_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb0d6141d0d40b199033b7441c3f659" is_pre_election: true
I20250815 01:54:59.324260 5766 raft_consensus.cc:2802] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 01:54:59.324517 5766 raft_consensus.cc:491] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:54:59.324426 5562 raft_consensus.cc:2466] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55cc1d7c8ccc40eaae8f854d4095e4b6 in term 0.
I20250815 01:54:59.324756 5766 raft_consensus.cc:3058] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 0 FOLLOWER]: Advancing to term 1
W20250815 01:54:59.327020 5608 tablet.cc:2378] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:54:59.330305 5766 raft_consensus.cc:513] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
I20250815 01:54:59.331765 5766 leader_election.cc:290] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [CANDIDATE]: Term 1 election: Requested vote from peers e752c8b174fa4c9ba4991e5a3fa9416c (127.0.106.193:43011), adb0d6141d0d40b199033b7441c3f659 (127.0.106.196:34343)
I20250815 01:54:59.332448 5161 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e34a99ab9646d6ab15e3de87215077" candidate_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c"
I20250815 01:54:59.332670 5562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "13e34a99ab9646d6ab15e3de87215077" candidate_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb0d6141d0d40b199033b7441c3f659"
I20250815 01:54:59.332969 5161 raft_consensus.cc:3058] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:54:59.333187 5562 raft_consensus.cc:3058] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:54:59.340457 5562 raft_consensus.cc:2466] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55cc1d7c8ccc40eaae8f854d4095e4b6 in term 1.
I20250815 01:54:59.340703 5161 raft_consensus.cc:2466] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55cc1d7c8ccc40eaae8f854d4095e4b6 in term 1.
I20250815 01:54:59.341425 5230 leader_election.cc:304] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55cc1d7c8ccc40eaae8f854d4095e4b6, adb0d6141d0d40b199033b7441c3f659; no voters:
I20250815 01:54:59.342135 5766 raft_consensus.cc:2802] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:54:59.344894 5766 raft_consensus.cc:695] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [term 1 LEADER]: Becoming Leader. State: Replica: 55cc1d7c8ccc40eaae8f854d4095e4b6, State: Running, Role: LEADER
I20250815 01:54:59.345863 5766 consensus_queue.cc:237] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } }
W20250815 01:54:59.357586 5207 tablet.cc:2378] T 13e34a99ab9646d6ab15e3de87215077 P e752c8b174fa4c9ba4991e5a3fa9416c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:54:59.358078 5018 catalog_manager.cc:5582] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55cc1d7c8ccc40eaae8f854d4095e4b6 (127.0.106.194). New cstate: current_term: 1 leader_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e752c8b174fa4c9ba4991e5a3fa9416c" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55cc1d7c8ccc40eaae8f854d4095e4b6" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 45325 } health_report { overall_health: HEALTHY } } }
I20250815 01:54:59.400450 427 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250815 01:54:59.403915 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e752c8b174fa4c9ba4991e5a3fa9416c to finish bootstrapping
I20250815 01:54:59.417441 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 55cc1d7c8ccc40eaae8f854d4095e4b6 to finish bootstrapping
I20250815 01:54:59.428224 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver adb0d6141d0d40b199033b7441c3f659 to finish bootstrapping
I20250815 01:54:59.439662 427 test_util.cc:276] Using random seed: -1385416680
I20250815 01:54:59.466035 427 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250815 01:54:59.466993 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 5077
I20250815 01:54:59.497826 5562 raft_consensus.cc:1273] T 13e34a99ab9646d6ab15e3de87215077 P adb0d6141d0d40b199033b7441c3f659 [term 1 FOLLOWER]: Refusing update from remote peer 55cc1d7c8ccc40eaae8f854d4095e4b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250815 01:54:59.498776 5230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.193:43011: connect: Connection refused (error 111)
I20250815 01:54:59.501822 5771 consensus_queue.cc:1035] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adb0d6141d0d40b199033b7441c3f659" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 01:54:59.503161 5230 consensus_peers.cc:489] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 -> Peer e752c8b174fa4c9ba4991e5a3fa9416c (127.0.106.193:43011): Couldn't send request to peer e752c8b174fa4c9ba4991e5a3fa9416c. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.193:43011: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 01:54:59.521606 5784 mvcc.cc:204] Tried to move back new op lower bound from 7189392996325220352 to 7189392995728179200. Current Snapshot: MvccSnapshot[applied={T|T < 7189392996325220352}]
I20250815 01:55:00.029261 5741 heartbeater.cc:499] Master 127.0.106.254:32789 was elected leader, sending a full tablet report...
W20250815 01:55:01.831509 5230 consensus_peers.cc:489] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 -> Peer e752c8b174fa4c9ba4991e5a3fa9416c (127.0.106.193:43011): Couldn't send request to peer e752c8b174fa4c9ba4991e5a3fa9416c. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.193:43011: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250815 01:55:02.315479 5408 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 01:55:02.343919 5274 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 01:55:02.349152 5542 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 01:55:02.391330 5676 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 01:55:04.108950 5408 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 01:55:04.115819 5274 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 01:55:04.150470 5542 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 01:55:04.193316 5676 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250815 01:55:04.513830 5230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.193:43011: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250815 01:55:04.523298 5230 consensus_peers.cc:489] T 13e34a99ab9646d6ab15e3de87215077 P 55cc1d7c8ccc40eaae8f854d4095e4b6 -> Peer e752c8b174fa4c9ba4991e5a3fa9416c (127.0.106.193:43011): Couldn't send request to peer e752c8b174fa4c9ba4991e5a3fa9416c. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.193:43011: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250815 01:55:06.630162 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 5210
I20250815 01:55:06.668293 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 5343
I20250815 01:55:06.693086 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 5477
I20250815 01:55:06.728986 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 5611
I20250815 01:55:06.754096 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 4985
2025-08-15T01:55:06Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (20909 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (20909 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250815 01:55:06.816802 427 test_util.cc:276] Using random seed: -1378039546
I20250815 01:55:06.820973 427 ts_itest-base.cc:115] Starting cluster with:
I20250815 01:55:06.821154 427 ts_itest-base.cc:116] --------------
I20250815 01:55:06.821317 427 ts_itest-base.cc:117] 1 tablet servers
I20250815 01:55:06.821457 427 ts_itest-base.cc:118] 1 replicas per TS
I20250815 01:55:06.821592 427 ts_itest-base.cc:119] --------------
2025-08-15T01:55:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T01:55:06Z Disabled control of system clock
I20250815 01:55:06.866663 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:43585
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:43585 with env {}
W20250815 01:55:07.177699 5889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:07.178270 5889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:07.178756 5889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:07.210546 5889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:55:07.210832 5889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:07.211038 5889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:55:07.211241 5889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:55:07.247771 5889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:43585
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:07.249013 5889 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:07.250628 5889 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:07.260679 5895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:07.262420 5896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:08.459216 5897 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1192 milliseconds
W20250815 01:55:08.461242 5898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:08.463531 5889 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.202s user 0.000s sys 0.010s
W20250815 01:55:08.463793 5889 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.202s user 0.000s sys 0.010s
I20250815 01:55:08.464049 5889 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:08.465124 5889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:08.468273 5889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:08.469707 5889 hybrid_clock.cc:648] HybridClock initialized: now 1755222908469646 us; error 78 us; skew 500 ppm
I20250815 01:55:08.470592 5889 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:08.478559 5889 webserver.cc:480] Webserver started at http://127.0.106.254:36947/ using document root <none> and password file <none>
I20250815 01:55:08.479566 5889 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:08.479794 5889 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:08.480297 5889 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:55:08.484880 5889 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d7a01ddf676749369265126e0a1997ae"
format_stamp: "Formatted at 2025-08-15 01:55:08 on dist-test-slave-h491"
I20250815 01:55:08.486030 5889 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d7a01ddf676749369265126e0a1997ae"
format_stamp: "Formatted at 2025-08-15 01:55:08 on dist-test-slave-h491"
I20250815 01:55:08.494390 5889 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250815 01:55:08.501061 5905 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:08.502235 5889 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250815 01:55:08.502599 5889 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d7a01ddf676749369265126e0a1997ae"
format_stamp: "Formatted at 2025-08-15 01:55:08 on dist-test-slave-h491"
I20250815 01:55:08.502929 5889 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:08.569850 5889 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:08.571868 5889 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:08.572306 5889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:08.644831 5889 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:43585
I20250815 01:55:08.644902 5956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:43585 every 8 connection(s)
I20250815 01:55:08.647557 5889 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 01:55:08.652901 5957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:08.657967 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 5889
I20250815 01:55:08.658574 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 01:55:08.678473 5957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae: Bootstrap starting.
I20250815 01:55:08.684402 5957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:08.686636 5957 log.cc:826] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:08.691537 5957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae: No bootstrap required, opened a new log
I20250815 01:55:08.711588 5957 raft_consensus.cc:357] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a01ddf676749369265126e0a1997ae" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 43585 } }
I20250815 01:55:08.712390 5957 raft_consensus.cc:383] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:08.712646 5957 raft_consensus.cc:738] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a01ddf676749369265126e0a1997ae, State: Initialized, Role: FOLLOWER
I20250815 01:55:08.713341 5957 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a01ddf676749369265126e0a1997ae" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 43585 } }
I20250815 01:55:08.713868 5957 raft_consensus.cc:397] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:55:08.714115 5957 raft_consensus.cc:491] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:55:08.714432 5957 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:08.718849 5957 raft_consensus.cc:513] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a01ddf676749369265126e0a1997ae" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 43585 } }
I20250815 01:55:08.719857 5957 leader_election.cc:304] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7a01ddf676749369265126e0a1997ae; no voters:
I20250815 01:55:08.721887 5957 leader_election.cc:290] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:55:08.722780 5962 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:55:08.725309 5962 raft_consensus.cc:695] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [term 1 LEADER]: Becoming Leader. State: Replica: d7a01ddf676749369265126e0a1997ae, State: Running, Role: LEADER
I20250815 01:55:08.726118 5962 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a01ddf676749369265126e0a1997ae" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 43585 } }
I20250815 01:55:08.727272 5957 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:55:08.735661 5963 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7a01ddf676749369265126e0a1997ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a01ddf676749369265126e0a1997ae" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 43585 } } }
I20250815 01:55:08.736405 5963 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [sys.catalog]: This master's current role is: LEADER
I20250815 01:55:08.737247 5964 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7a01ddf676749369265126e0a1997ae. Latest consensus state: current_term: 1 leader_uuid: "d7a01ddf676749369265126e0a1997ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7a01ddf676749369265126e0a1997ae" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 43585 } } }
I20250815 01:55:08.737937 5964 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae [sys.catalog]: This master's current role is: LEADER
I20250815 01:55:08.743383 5968 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:55:08.753780 5968 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:55:08.773984 5968 catalog_manager.cc:1349] Generated new cluster ID: 7233e93832084556a975612e37bdf458
I20250815 01:55:08.774339 5968 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:55:08.796922 5968 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:55:08.798445 5968 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:55:08.811313 5968 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d7a01ddf676749369265126e0a1997ae: Generated new TSK 0
I20250815 01:55:08.812261 5968 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 01:55:08.831205 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:43585
--builtin_ntp_servers=127.0.106.212:43681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 01:55:09.152192 5981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:09.152748 5981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:09.153343 5981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:09.186816 5981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:09.187680 5981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:55:09.224213 5981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:43681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:09.225561 5981 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:09.227164 5981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:09.240036 5987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:09.245282 5990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:09.242044 5988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:10.756809 5989 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1513 milliseconds
I20250815 01:55:10.757014 5981 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:10.761787 5981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:10.764632 5981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:10.766157 5981 hybrid_clock.cc:648] HybridClock initialized: now 1755222910766105 us; error 54 us; skew 500 ppm
I20250815 01:55:10.767279 5981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:10.775158 5981 webserver.cc:480] Webserver started at http://127.0.106.193:35539/ using document root <none> and password file <none>
I20250815 01:55:10.776484 5981 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:10.776921 5981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:10.777571 5981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:55:10.784380 5981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "705c07536d334a3cb1ead97dd9bfcf69"
format_stamp: "Formatted at 2025-08-15 01:55:10 on dist-test-slave-h491"
I20250815 01:55:10.786108 5981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "705c07536d334a3cb1ead97dd9bfcf69"
format_stamp: "Formatted at 2025-08-15 01:55:10 on dist-test-slave-h491"
I20250815 01:55:10.796002 5981 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20250815 01:55:10.804253 5997 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:10.805629 5981 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250815 01:55:10.806064 5981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "705c07536d334a3cb1ead97dd9bfcf69"
format_stamp: "Formatted at 2025-08-15 01:55:10 on dist-test-slave-h491"
I20250815 01:55:10.806588 5981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:10.900475 5981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:10.902590 5981 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:10.903218 5981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:10.906730 5981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:55:10.912503 5981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:55:10.912854 5981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:10.913134 5981 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:55:10.913304 5981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:11.087718 5981 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:38375
I20250815 01:55:11.087815 6109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:38375 every 8 connection(s)
I20250815 01:55:11.090870 5981 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 01:55:11.097606 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 5981
I20250815 01:55:11.098273 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755222754379616-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 01:55:11.118593 6110 heartbeater.cc:344] Connected to a master server at 127.0.106.254:43585
I20250815 01:55:11.119072 6110 heartbeater.cc:461] Registering TS with master...
I20250815 01:55:11.120344 6110 heartbeater.cc:507] Master 127.0.106.254:43585 requested a full tablet report, sending...
I20250815 01:55:11.123140 5921 ts_manager.cc:194] Registered new tserver with Master: 705c07536d334a3cb1ead97dd9bfcf69 (127.0.106.193:38375)
I20250815 01:55:11.125258 5921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:56519
I20250815 01:55:11.134778 427 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250815 01:55:11.175252 5921 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49798:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 01:55:11.255235 6045 tablet_service.cc:1468] Processing CreateTablet for tablet d57b2fd012fa40e5a14919ec46422ee5 (DEFAULT_TABLE table=TestTable [id=bf7ad88e632e42fdad0d649e917657bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:55:11.256856 6045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57b2fd012fa40e5a14919ec46422ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:11.278046 6125 tablet_bootstrap.cc:492] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69: Bootstrap starting.
I20250815 01:55:11.285874 6125 tablet_bootstrap.cc:654] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:11.288383 6125 log.cc:826] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:11.293977 6125 tablet_bootstrap.cc:492] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69: No bootstrap required, opened a new log
I20250815 01:55:11.294561 6125 ts_tablet_manager.cc:1397] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20250815 01:55:11.321180 6125 raft_consensus.cc:357] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705c07536d334a3cb1ead97dd9bfcf69" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38375 } }
I20250815 01:55:11.322022 6125 raft_consensus.cc:383] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:11.322319 6125 raft_consensus.cc:738] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705c07536d334a3cb1ead97dd9bfcf69, State: Initialized, Role: FOLLOWER
I20250815 01:55:11.323189 6125 consensus_queue.cc:260] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705c07536d334a3cb1ead97dd9bfcf69" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38375 } }
I20250815 01:55:11.323884 6125 raft_consensus.cc:397] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:55:11.324220 6125 raft_consensus.cc:491] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:55:11.324647 6125 raft_consensus.cc:3058] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:11.331800 6125 raft_consensus.cc:513] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705c07536d334a3cb1ead97dd9bfcf69" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38375 } }
I20250815 01:55:11.332800 6125 leader_election.cc:304] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705c07536d334a3cb1ead97dd9bfcf69; no voters:
I20250815 01:55:11.334697 6125 leader_election.cc:290] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:55:11.335012 6127 raft_consensus.cc:2802] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:55:11.338532 6110 heartbeater.cc:499] Master 127.0.106.254:43585 was elected leader, sending a full tablet report...
I20250815 01:55:11.338814 6125 ts_tablet_manager.cc:1428] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69: Time spent starting tablet: real 0.044s user 0.035s sys 0.009s
I20250815 01:55:11.339460 6127 raft_consensus.cc:695] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [term 1 LEADER]: Becoming Leader. State: Replica: 705c07536d334a3cb1ead97dd9bfcf69, State: Running, Role: LEADER
I20250815 01:55:11.340194 6127 consensus_queue.cc:237] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705c07536d334a3cb1ead97dd9bfcf69" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38375 } }
W20250815 01:55:11.346161 6111 tablet.cc:2378] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:55:11.355111 5921 catalog_manager.cc:5582] T d57b2fd012fa40e5a14919ec46422ee5 P 705c07536d334a3cb1ead97dd9bfcf69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 705c07536d334a3cb1ead97dd9bfcf69 (127.0.106.193). New cstate: current_term: 1 leader_uuid: "705c07536d334a3cb1ead97dd9bfcf69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705c07536d334a3cb1ead97dd9bfcf69" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38375 } health_report { overall_health: HEALTHY } } }
I20250815 01:55:11.386540 427 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250815 01:55:11.390095 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 705c07536d334a3cb1ead97dd9bfcf69 to finish bootstrapping
I20250815 01:55:14.094903 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 5981
I20250815 01:55:14.122071 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 5889
2025-08-15T01:55:14Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7366 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7366 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250815 01:55:14.183377 427 test_util.cc:276] Using random seed: -1370672972
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-15T01:55:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T01:55:14Z Disabled control of system clock
I20250815 01:55:14.238516 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:42735
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:42735 with env {}
W20250815 01:55:14.553000 6153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:14.553575 6153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:14.554100 6153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:14.586076 6153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:55:14.586431 6153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:14.586654 6153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:55:14.586862 6153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:55:14.623613 6153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:42735
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:42735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:14.624938 6153 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:14.626718 6153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:14.637547 6159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:14.639215 6160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:15.851606 6162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:15.853772 6161 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1210 milliseconds
I20250815 01:55:15.853950 6153 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:15.855232 6153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:15.858407 6153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:15.859853 6153 hybrid_clock.cc:648] HybridClock initialized: now 1755222915859809 us; error 56 us; skew 500 ppm
I20250815 01:55:15.860633 6153 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:15.867848 6153 webserver.cc:480] Webserver started at http://127.0.106.254:45053/ using document root <none> and password file <none>
I20250815 01:55:15.868839 6153 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:15.869102 6153 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:15.869608 6153 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:55:15.874046 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/instance:
uuid: "1c736de2342f4917a76307a83bd384fa"
format_stamp: "Formatted at 2025-08-15 01:55:15 on dist-test-slave-h491"
I20250815 01:55:15.875252 6153 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal/instance:
uuid: "1c736de2342f4917a76307a83bd384fa"
format_stamp: "Formatted at 2025-08-15 01:55:15 on dist-test-slave-h491"
I20250815 01:55:15.882920 6153 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250815 01:55:15.888835 6170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:15.889951 6153 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250815 01:55:15.890337 6153 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
uuid: "1c736de2342f4917a76307a83bd384fa"
format_stamp: "Formatted at 2025-08-15 01:55:15 on dist-test-slave-h491"
I20250815 01:55:15.890715 6153 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:15.940899 6153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:15.942528 6153 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:15.942986 6153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:16.013412 6153 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:42735
I20250815 01:55:16.013500 6221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:42735 every 8 connection(s)
I20250815 01:55:16.016206 6153 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/info.pb
I20250815 01:55:16.017127 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 6153
I20250815 01:55:16.017704 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal/instance
I20250815 01:55:16.023484 6222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:16.047984 6222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa: Bootstrap starting.
I20250815 01:55:16.053442 6222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:16.055228 6222 log.cc:826] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:16.060056 6222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa: No bootstrap required, opened a new log
I20250815 01:55:16.078505 6222 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c736de2342f4917a76307a83bd384fa" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } }
I20250815 01:55:16.079421 6222 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:16.079728 6222 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c736de2342f4917a76307a83bd384fa, State: Initialized, Role: FOLLOWER
I20250815 01:55:16.080610 6222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c736de2342f4917a76307a83bd384fa" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } }
I20250815 01:55:16.081120 6222 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:55:16.081424 6222 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:55:16.081765 6222 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:16.086009 6222 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c736de2342f4917a76307a83bd384fa" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } }
I20250815 01:55:16.086828 6222 leader_election.cc:304] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c736de2342f4917a76307a83bd384fa; no voters:
I20250815 01:55:16.088512 6222 leader_election.cc:290] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:55:16.089352 6227 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:55:16.092351 6227 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [term 1 LEADER]: Becoming Leader. State: Replica: 1c736de2342f4917a76307a83bd384fa, State: Running, Role: LEADER
I20250815 01:55:16.093119 6222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:55:16.093207 6227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c736de2342f4917a76307a83bd384fa" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } }
I20250815 01:55:16.104180 6228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c736de2342f4917a76307a83bd384fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c736de2342f4917a76307a83bd384fa" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } } }
I20250815 01:55:16.104943 6228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [sys.catalog]: This master's current role is: LEADER
I20250815 01:55:16.105401 6229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c736de2342f4917a76307a83bd384fa. Latest consensus state: current_term: 1 leader_uuid: "1c736de2342f4917a76307a83bd384fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c736de2342f4917a76307a83bd384fa" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } } }
I20250815 01:55:16.106098 6229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa [sys.catalog]: This master's current role is: LEADER
I20250815 01:55:16.108103 6237 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:55:16.118288 6237 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:55:16.135535 6237 catalog_manager.cc:1349] Generated new cluster ID: da792e1d792f471b8058f2c18595ff35
I20250815 01:55:16.135807 6237 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:55:16.159338 6237 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:55:16.160804 6237 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:55:16.181715 6237 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1c736de2342f4917a76307a83bd384fa: Generated new TSK 0
I20250815 01:55:16.182972 6237 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 01:55:16.202827 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:42735
--builtin_ntp_servers=127.0.106.212:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250815 01:55:16.518744 6246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:16.519326 6246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:16.519846 6246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:16.552127 6246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:16.552979 6246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:55:16.588516 6246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:42735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:16.589825 6246 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:16.591521 6246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:16.604959 6252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:16.606915 6253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:17.810974 6254 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 01:55:17.812543 6255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:55:17.812633 6246 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:17.816061 6246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:17.818917 6246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:17.820363 6246 hybrid_clock.cc:648] HybridClock initialized: now 1755222917820318 us; error 50 us; skew 500 ppm
I20250815 01:55:17.821163 6246 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:17.828526 6246 webserver.cc:480] Webserver started at http://127.0.106.193:34405/ using document root <none> and password file <none>
I20250815 01:55:17.829524 6246 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:17.829730 6246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:17.830224 6246 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:55:17.834754 6246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/instance:
uuid: "4663eeed1fcb45768c59c9141eb245ba"
format_stamp: "Formatted at 2025-08-15 01:55:17 on dist-test-slave-h491"
I20250815 01:55:17.835887 6246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal/instance:
uuid: "4663eeed1fcb45768c59c9141eb245ba"
format_stamp: "Formatted at 2025-08-15 01:55:17 on dist-test-slave-h491"
I20250815 01:55:17.843757 6246 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250815 01:55:17.850090 6262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:17.851311 6246 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250815 01:55:17.851639 6246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
uuid: "4663eeed1fcb45768c59c9141eb245ba"
format_stamp: "Formatted at 2025-08-15 01:55:17 on dist-test-slave-h491"
I20250815 01:55:17.851974 6246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:17.925680 6246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:17.927203 6246 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:17.927639 6246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:17.930121 6246 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:55:17.934254 6246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:55:17.934465 6246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250815 01:55:17.934725 6246 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:55:17.934900 6246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:18.078930 6246 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:44053
I20250815 01:55:18.079041 6374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:44053 every 8 connection(s)
I20250815 01:55:18.081511 6246 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/info.pb
I20250815 01:55:18.085063 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 6246
I20250815 01:55:18.085511 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal/instance
I20250815 01:55:18.092235 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:42735
--builtin_ntp_servers=127.0.106.212:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 01:55:18.105226 6375 heartbeater.cc:344] Connected to a master server at 127.0.106.254:42735
I20250815 01:55:18.105700 6375 heartbeater.cc:461] Registering TS with master...
I20250815 01:55:18.106822 6375 heartbeater.cc:507] Master 127.0.106.254:42735 requested a full tablet report, sending...
I20250815 01:55:18.109521 6187 ts_manager.cc:194] Registered new tserver with Master: 4663eeed1fcb45768c59c9141eb245ba (127.0.106.193:44053)
I20250815 01:55:18.111690 6187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:38911
W20250815 01:55:18.401743 6379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:18.402284 6379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:18.402856 6379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:18.434511 6379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:18.435405 6379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:55:18.470886 6379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:42735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:18.472200 6379 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:18.473847 6379 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:18.486039 6385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:55:19.116014 6375 heartbeater.cc:499] Master 127.0.106.254:42735 was elected leader, sending a full tablet report...
W20250815 01:55:18.487811 6386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:19.695976 6388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:19.699657 6379 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.213s user 0.357s sys 0.851s
W20250815 01:55:19.699959 6379 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.213s user 0.357s sys 0.851s
W20250815 01:55:19.700196 6387 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1212 milliseconds
I20250815 01:55:19.700191 6379 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:19.701574 6379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:19.703817 6379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:19.705155 6379 hybrid_clock.cc:648] HybridClock initialized: now 1755222919705124 us; error 49 us; skew 500 ppm
I20250815 01:55:19.705957 6379 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:19.713793 6379 webserver.cc:480] Webserver started at http://127.0.106.194:42559/ using document root <none> and password file <none>
I20250815 01:55:19.714903 6379 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:19.715145 6379 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:19.715591 6379 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:55:19.720094 6379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/instance:
uuid: "7072378e90f94886a6c5ef09dff30b3d"
format_stamp: "Formatted at 2025-08-15 01:55:19 on dist-test-slave-h491"
I20250815 01:55:19.721225 6379 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal/instance:
uuid: "7072378e90f94886a6c5ef09dff30b3d"
format_stamp: "Formatted at 2025-08-15 01:55:19 on dist-test-slave-h491"
I20250815 01:55:19.729009 6379 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.001s
I20250815 01:55:19.735189 6395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:19.736374 6379 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 01:55:19.736721 6379 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
uuid: "7072378e90f94886a6c5ef09dff30b3d"
format_stamp: "Formatted at 2025-08-15 01:55:19 on dist-test-slave-h491"
I20250815 01:55:19.737061 6379 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:19.811826 6379 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:19.813298 6379 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:19.813724 6379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:19.816227 6379 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:55:19.820351 6379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:55:19.820557 6379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:19.820765 6379 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:55:19.820899 6379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:19.955857 6379 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:33735
I20250815 01:55:19.955962 6507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:33735 every 8 connection(s)
I20250815 01:55:19.958521 6379 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/info.pb
I20250815 01:55:19.963289 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 6379
I20250815 01:55:19.963840 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal/instance
I20250815 01:55:19.970839 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:42735
--builtin_ntp_servers=127.0.106.212:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 01:55:19.985178 6508 heartbeater.cc:344] Connected to a master server at 127.0.106.254:42735
I20250815 01:55:19.985656 6508 heartbeater.cc:461] Registering TS with master...
I20250815 01:55:19.986793 6508 heartbeater.cc:507] Master 127.0.106.254:42735 requested a full tablet report, sending...
I20250815 01:55:19.989439 6187 ts_manager.cc:194] Registered new tserver with Master: 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735)
I20250815 01:55:19.991515 6187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:39497
W20250815 01:55:20.282310 6512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:20.282866 6512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:20.283306 6512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:20.314836 6512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:20.315629 6512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:55:20.350579 6512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:42735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:20.351859 6512 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:20.353397 6512 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:20.364730 6518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:55:20.995560 6508 heartbeater.cc:499] Master 127.0.106.254:42735 was elected leader, sending a full tablet report...
W20250815 01:55:20.366550 6519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:21.573781 6521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:21.576675 6520 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1205 milliseconds
I20250815 01:55:21.576820 6512 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:21.578001 6512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:21.581696 6512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:21.583132 6512 hybrid_clock.cc:648] HybridClock initialized: now 1755222921583072 us; error 73 us; skew 500 ppm
I20250815 01:55:21.583953 6512 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:21.589951 6512 webserver.cc:480] Webserver started at http://127.0.106.195:40377/ using document root <none> and password file <none>
I20250815 01:55:21.590924 6512 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:21.591120 6512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:21.591625 6512 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:55:21.596065 6512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/instance:
uuid: "a4df3b043e894188b087c5b7a505dcf8"
format_stamp: "Formatted at 2025-08-15 01:55:21 on dist-test-slave-h491"
I20250815 01:55:21.597117 6512 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal/instance:
uuid: "a4df3b043e894188b087c5b7a505dcf8"
format_stamp: "Formatted at 2025-08-15 01:55:21 on dist-test-slave-h491"
I20250815 01:55:21.604171 6512 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.008s
I20250815 01:55:21.609946 6528 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:21.611017 6512 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 01:55:21.611338 6512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
uuid: "a4df3b043e894188b087c5b7a505dcf8"
format_stamp: "Formatted at 2025-08-15 01:55:21 on dist-test-slave-h491"
I20250815 01:55:21.611693 6512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:21.668479 6512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:21.669981 6512 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:21.670447 6512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:21.672966 6512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:55:21.677188 6512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 01:55:21.677395 6512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:21.677589 6512 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 01:55:21.677719 6512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:21.812376 6512 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:44785
I20250815 01:55:21.812506 6640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:44785 every 8 connection(s)
I20250815 01:55:21.814922 6512 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/info.pb
I20250815 01:55:21.817126 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 6512
I20250815 01:55:21.817654 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal/instance
I20250815 01:55:21.836951 6641 heartbeater.cc:344] Connected to a master server at 127.0.106.254:42735
I20250815 01:55:21.837455 6641 heartbeater.cc:461] Registering TS with master...
I20250815 01:55:21.838651 6641 heartbeater.cc:507] Master 127.0.106.254:42735 requested a full tablet report, sending...
I20250815 01:55:21.840723 6186 ts_manager.cc:194] Registered new tserver with Master: a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785)
I20250815 01:55:21.842070 6186 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:47347
I20250815 01:55:21.852602 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:55:21.881714 427 test_util.cc:276] Using random seed: -1362974628
I20250815 01:55:21.921882 6186 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46324:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250815 01:55:21.924263 6186 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 01:55:21.983731 6443 tablet_service.cc:1468] Processing CreateTablet for tablet 5c9445a6b425407eb992dd737ab6b654 (DEFAULT_TABLE table=pre_rebuild [id=d74db82f60004599b8b9ecfd0725d6b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:55:21.985900 6443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9445a6b425407eb992dd737ab6b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:21.992000 6310 tablet_service.cc:1468] Processing CreateTablet for tablet 5c9445a6b425407eb992dd737ab6b654 (DEFAULT_TABLE table=pre_rebuild [id=d74db82f60004599b8b9ecfd0725d6b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:55:21.994663 6310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9445a6b425407eb992dd737ab6b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:21.991953 6576 tablet_service.cc:1468] Processing CreateTablet for tablet 5c9445a6b425407eb992dd737ab6b654 (DEFAULT_TABLE table=pre_rebuild [id=d74db82f60004599b8b9ecfd0725d6b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:55:21.997305 6576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c9445a6b425407eb992dd737ab6b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:22.031919 6667 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Bootstrap starting.
I20250815 01:55:22.034920 6666 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Bootstrap starting.
I20250815 01:55:22.039870 6665 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Bootstrap starting.
I20250815 01:55:22.041923 6667 tablet_bootstrap.cc:654] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:22.043432 6666 tablet_bootstrap.cc:654] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:22.044395 6667 log.cc:826] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:22.045914 6666 log.cc:826] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:22.051730 6665 tablet_bootstrap.cc:654] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:22.054273 6665 log.cc:826] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:22.056497 6667 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: No bootstrap required, opened a new log
I20250815 01:55:22.057160 6667 ts_tablet_manager.cc:1397] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.009s
I20250815 01:55:22.067987 6666 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: No bootstrap required, opened a new log
I20250815 01:55:22.068619 6666 ts_tablet_manager.cc:1397] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.010s
I20250815 01:55:22.069782 6665 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: No bootstrap required, opened a new log
I20250815 01:55:22.070335 6665 ts_tablet_manager.cc:1397] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.018s
I20250815 01:55:22.089349 6667 raft_consensus.cc:357] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.090528 6667 raft_consensus.cc:383] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:22.090906 6667 raft_consensus.cc:738] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4663eeed1fcb45768c59c9141eb245ba, State: Initialized, Role: FOLLOWER
I20250815 01:55:22.092995 6667 consensus_queue.cc:260] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.096328 6666 raft_consensus.cc:357] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.097249 6666 raft_consensus.cc:383] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:22.097568 6666 raft_consensus.cc:738] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4df3b043e894188b087c5b7a505dcf8, State: Initialized, Role: FOLLOWER
I20250815 01:55:22.098778 6666 consensus_queue.cc:260] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.104058 6667 ts_tablet_manager.cc:1428] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Time spent starting tablet: real 0.047s user 0.034s sys 0.003s
I20250815 01:55:22.106232 6641 heartbeater.cc:499] Master 127.0.106.254:42735 was elected leader, sending a full tablet report...
I20250815 01:55:22.106745 6666 ts_tablet_manager.cc:1428] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Time spent starting tablet: real 0.038s user 0.031s sys 0.004s
I20250815 01:55:22.125947 6665 raft_consensus.cc:357] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.126691 6665 raft_consensus.cc:383] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:22.126952 6665 raft_consensus.cc:738] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7072378e90f94886a6c5ef09dff30b3d, State: Initialized, Role: FOLLOWER
I20250815 01:55:22.127651 6665 consensus_queue.cc:260] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.131124 6665 ts_tablet_manager.cc:1428] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Time spent starting tablet: real 0.060s user 0.028s sys 0.002s
W20250815 01:55:22.216547 6509 tablet.cc:2378] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 01:55:22.321660 6642 tablet.cc:2378] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 01:55:22.341400 6376 tablet.cc:2378] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:55:22.362677 6671 raft_consensus.cc:491] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:55:22.363170 6671 raft_consensus.cc:513] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.365520 6671 leader_election.cc:290] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735), a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785)
I20250815 01:55:22.376385 6596 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4df3b043e894188b087c5b7a505dcf8" is_pre_election: true
I20250815 01:55:22.376384 6463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7072378e90f94886a6c5ef09dff30b3d" is_pre_election: true
I20250815 01:55:22.377247 6596 raft_consensus.cc:2466] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4663eeed1fcb45768c59c9141eb245ba in term 0.
I20250815 01:55:22.377250 6463 raft_consensus.cc:2466] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4663eeed1fcb45768c59c9141eb245ba in term 0.
I20250815 01:55:22.378665 6266 leader_election.cc:304] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4663eeed1fcb45768c59c9141eb245ba, a4df3b043e894188b087c5b7a505dcf8; no voters:
I20250815 01:55:22.379717 6671 raft_consensus.cc:2802] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 01:55:22.380213 6671 raft_consensus.cc:491] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:55:22.380650 6671 raft_consensus.cc:3058] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:22.387224 6671 raft_consensus.cc:513] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.388749 6671 leader_election.cc:290] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 1 election: Requested vote from peers 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735), a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785)
I20250815 01:55:22.389441 6463 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7072378e90f94886a6c5ef09dff30b3d"
I20250815 01:55:22.389674 6596 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4df3b043e894188b087c5b7a505dcf8"
I20250815 01:55:22.389858 6463 raft_consensus.cc:3058] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:22.390142 6596 raft_consensus.cc:3058] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:22.394243 6463 raft_consensus.cc:2466] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4663eeed1fcb45768c59c9141eb245ba in term 1.
I20250815 01:55:22.394448 6596 raft_consensus.cc:2466] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4663eeed1fcb45768c59c9141eb245ba in term 1.
I20250815 01:55:22.395319 6266 leader_election.cc:304] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4663eeed1fcb45768c59c9141eb245ba, a4df3b043e894188b087c5b7a505dcf8; no voters:
I20250815 01:55:22.395906 6671 raft_consensus.cc:2802] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:55:22.397440 6671 raft_consensus.cc:695] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 LEADER]: Becoming Leader. State: Replica: 4663eeed1fcb45768c59c9141eb245ba, State: Running, Role: LEADER
I20250815 01:55:22.398324 6671 consensus_queue.cc:237] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:22.407573 6185 catalog_manager.cc:5582] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 4663eeed1fcb45768c59c9141eb245ba (127.0.106.193). New cstate: current_term: 1 leader_uuid: "4663eeed1fcb45768c59c9141eb245ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } health_report { overall_health: UNKNOWN } } }
I20250815 01:55:22.535868 6596 raft_consensus.cc:1273] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 1 FOLLOWER]: Refusing update from remote peer 4663eeed1fcb45768c59c9141eb245ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 01:55:22.536089 6463 raft_consensus.cc:1273] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 1 FOLLOWER]: Refusing update from remote peer 4663eeed1fcb45768c59c9141eb245ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 01:55:22.537334 6671 consensus_queue.cc:1035] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:55:22.538215 6676 consensus_queue.cc:1035] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:55:22.568727 6686 mvcc.cc:204] Tried to move back new op lower bound from 7189393090690801664 to 7189393090150473728. Current Snapshot: MvccSnapshot[applied={T|T < 7189393090690801664}]
W20250815 01:55:22.684827 6264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.106.194:33735, user_credentials={real_user=slave}} blocked reactor thread for 74673.8us
I20250815 01:55:27.766402 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 6153
W20250815 01:55:28.194012 6718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:28.194653 6718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:28.226274 6718 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 01:55:28.668191 6508 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:42735 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:42735: connect: Connection refused (error 111)
W20250815 01:55:28.683236 6375 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:42735 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:42735: connect: Connection refused (error 111)
W20250815 01:55:28.684172 6641 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:42735 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:42735: connect: Connection refused (error 111)
W20250815 01:55:29.672387 6723 debug-util.cc:398] Leaking SignalData structure 0x7b08000373e0 after lost signal to thread 6718
W20250815 01:55:29.672870 6723 kernel_stack_watchdog.cc:198] Thread 6718 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:55:29.749830 6718 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.481s user 0.001s sys 0.001s
W20250815 01:55:29.872586 6718 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.604s user 0.005s sys 0.009s
I20250815 01:55:29.956539 6718 minidump.cc:252] Setting minidump size limit to 20M
I20250815 01:55:29.958467 6718 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:29.959556 6718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:29.969883 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:29.972136 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:29.975862 6756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:55:29.976392 6718 server_base.cc:1047] running on GCE node
I20250815 01:55:29.977355 6718 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250815 01:55:29.977787 6718 hybrid_clock.cc:648] HybridClock initialized: now 1755222929977762 us; error 132016 us; skew 500 ppm
I20250815 01:55:29.978526 6718 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:29.983053 6718 webserver.cc:480] Webserver started at http://0.0.0.0:40155/ using document root <none> and password file <none>
I20250815 01:55:29.983914 6718 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:29.984139 6718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:29.984580 6718 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 01:55:29.989164 6718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/instance:
uuid: "95c8e4b59e8942aba1b04c3b909d6458"
format_stamp: "Formatted at 2025-08-15 01:55:29 on dist-test-slave-h491"
I20250815 01:55:29.990281 6718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal/instance:
uuid: "95c8e4b59e8942aba1b04c3b909d6458"
format_stamp: "Formatted at 2025-08-15 01:55:29 on dist-test-slave-h491"
I20250815 01:55:29.996448 6718 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250815 01:55:30.001565 6760 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:30.002519 6718 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250815 01:55:30.002857 6718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
uuid: "95c8e4b59e8942aba1b04c3b909d6458"
format_stamp: "Formatted at 2025-08-15 01:55:29 on dist-test-slave-h491"
I20250815 01:55:30.003181 6718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:30.141819 6718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:30.143306 6718 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:30.143731 6718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:30.148243 6718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:30.162460 6718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Bootstrap starting.
I20250815 01:55:30.167300 6718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:30.168907 6718 log.cc:826] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:30.172966 6718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: No bootstrap required, opened a new log
I20250815 01:55:30.189669 6718 raft_consensus.cc:357] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER }
I20250815 01:55:30.190174 6718 raft_consensus.cc:383] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:30.190459 6718 raft_consensus.cc:738] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c8e4b59e8942aba1b04c3b909d6458, State: Initialized, Role: FOLLOWER
I20250815 01:55:30.191139 6718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER }
I20250815 01:55:30.191586 6718 raft_consensus.cc:397] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:55:30.191851 6718 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:55:30.192155 6718 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:30.196064 6718 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER }
I20250815 01:55:30.196707 6718 leader_election.cc:304] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95c8e4b59e8942aba1b04c3b909d6458; no voters:
I20250815 01:55:30.198343 6718 leader_election.cc:290] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 01:55:30.198623 6767 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:55:30.201412 6767 raft_consensus.cc:695] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 LEADER]: Becoming Leader. State: Replica: 95c8e4b59e8942aba1b04c3b909d6458, State: Running, Role: LEADER
I20250815 01:55:30.202270 6767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER }
I20250815 01:55:30.209846 6768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95c8e4b59e8942aba1b04c3b909d6458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER } }
I20250815 01:55:30.210554 6768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: This master's current role is: LEADER
I20250815 01:55:30.210728 6769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c8e4b59e8942aba1b04c3b909d6458. Latest consensus state: current_term: 1 leader_uuid: "95c8e4b59e8942aba1b04c3b909d6458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER } }
I20250815 01:55:30.211151 6769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: This master's current role is: LEADER
I20250815 01:55:30.221591 6718 tablet_replica.cc:331] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: stopping tablet replica
I20250815 01:55:30.222115 6718 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 LEADER]: Raft consensus shutting down.
I20250815 01:55:30.222534 6718 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250815 01:55:30.224606 6718 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250815 01:55:30.225021 6718 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250815 01:55:30.279140 6718 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250815 01:55:31.308887 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 6246
I20250815 01:55:31.357203 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 6379
I20250815 01:55:31.391201 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 6512
I20250815 01:55:31.427572 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:42735
--webserver_interface=127.0.106.254
--webserver_port=45053
--builtin_ntp_servers=127.0.106.212:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:42735 with env {}
W20250815 01:55:31.746589 6777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:31.747197 6777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:31.747687 6777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:31.779901 6777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 01:55:31.780267 6777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:31.780526 6777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 01:55:31.780766 6777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 01:55:31.817261 6777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:42735
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:42735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=45053
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:31.818706 6777 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:31.820330 6777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:31.832218 6783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:33.235486 6782 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 6777
W20250815 01:55:31.833240 6784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:33.652546 6782 kernel_stack_watchdog.cc:198] Thread 6777 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:55:33.657809 6777 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.825s user 0.001s sys 0.001s
W20250815 01:55:33.657786 6785 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1819 milliseconds
W20250815 01:55:33.658190 6777 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.826s user 0.001s sys 0.001s
W20250815 01:55:33.659696 6786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:55:33.659785 6777 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:33.660998 6777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:33.663416 6777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:33.664806 6777 hybrid_clock.cc:648] HybridClock initialized: now 1755222933664755 us; error 49 us; skew 500 ppm
I20250815 01:55:33.665601 6777 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:33.671705 6777 webserver.cc:480] Webserver started at http://127.0.106.254:45053/ using document root <none> and password file <none>
I20250815 01:55:33.672619 6777 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:33.672852 6777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:33.680835 6777 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20250815 01:55:33.685599 6793 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:33.686818 6777 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 01:55:33.687160 6777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
uuid: "95c8e4b59e8942aba1b04c3b909d6458"
format_stamp: "Formatted at 2025-08-15 01:55:29 on dist-test-slave-h491"
I20250815 01:55:33.689093 6777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:33.739959 6777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:33.741519 6777 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:33.741961 6777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:33.815207 6777 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:42735
I20250815 01:55:33.815275 6844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:42735 every 8 connection(s)
I20250815 01:55:33.817988 6777 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/info.pb
I20250815 01:55:33.823613 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 6777
I20250815 01:55:33.825162 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:44053
--local_ip_for_outbound_sockets=127.0.106.193
--tserver_master_addrs=127.0.106.254:42735
--webserver_port=34405
--webserver_interface=127.0.106.193
--builtin_ntp_servers=127.0.106.212:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 01:55:33.828450 6845 sys_catalog.cc:263] Verifying existing consensus state
I20250815 01:55:33.846112 6845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Bootstrap starting.
I20250815 01:55:33.861826 6845 log.cc:826] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:33.877213 6845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:55:33.878240 6845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Bootstrap complete.
I20250815 01:55:33.907248 6845 raft_consensus.cc:357] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } }
I20250815 01:55:33.908063 6845 raft_consensus.cc:738] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c8e4b59e8942aba1b04c3b909d6458, State: Initialized, Role: FOLLOWER
I20250815 01:55:33.908896 6845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } }
I20250815 01:55:33.909655 6845 raft_consensus.cc:397] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 01:55:33.909911 6845 raft_consensus.cc:491] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 01:55:33.910233 6845 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:55:33.914425 6845 raft_consensus.cc:513] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } }
I20250815 01:55:33.915103 6845 leader_election.cc:304] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95c8e4b59e8942aba1b04c3b909d6458; no voters:
I20250815 01:55:33.917325 6845 leader_election.cc:290] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 01:55:33.917870 6849 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 01:55:33.921276 6849 raft_consensus.cc:695] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [term 2 LEADER]: Becoming Leader. State: Replica: 95c8e4b59e8942aba1b04c3b909d6458, State: Running, Role: LEADER
I20250815 01:55:33.922243 6849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } }
I20250815 01:55:33.923082 6845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 01:55:33.932829 6850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "95c8e4b59e8942aba1b04c3b909d6458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } } }
I20250815 01:55:33.933593 6850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: This master's current role is: LEADER
I20250815 01:55:33.935690 6851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c8e4b59e8942aba1b04c3b909d6458. Latest consensus state: current_term: 2 leader_uuid: "95c8e4b59e8942aba1b04c3b909d6458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c8e4b59e8942aba1b04c3b909d6458" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 42735 } } }
I20250815 01:55:33.936321 6851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458 [sys.catalog]: This master's current role is: LEADER
I20250815 01:55:33.944914 6856 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 01:55:33.959734 6856 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=3f0c8ddfecad442ebd5cad60f5a8a6f0]
I20250815 01:55:33.966806 6856 tablet_loader.cc:96] loaded metadata for tablet 5c9445a6b425407eb992dd737ab6b654 (table pre_rebuild [id=3f0c8ddfecad442ebd5cad60f5a8a6f0])
I20250815 01:55:33.968250 6856 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 01:55:33.988605 6856 catalog_manager.cc:1349] Generated new cluster ID: d5aac08ad5664d728558b189991390aa
I20250815 01:55:33.988984 6856 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 01:55:34.054879 6856 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 01:55:34.056351 6856 catalog_manager.cc:1506] Loading token signing keys...
I20250815 01:55:34.070560 6856 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Generated new TSK 0
I20250815 01:55:34.071461 6856 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 01:55:34.227620 6847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:34.228148 6847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:34.228649 6847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:34.262017 6847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:34.262959 6847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250815 01:55:34.298324 6847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:44053
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=34405
--tserver_master_addrs=127.0.106.254:42735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:34.299695 6847 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:34.301304 6847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:34.315847 6873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:34.319506 6876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:34.316489 6874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:35.553125 6875 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1230 milliseconds
I20250815 01:55:35.553229 6847 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:35.554528 6847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:35.557245 6847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:35.558737 6847 hybrid_clock.cc:648] HybridClock initialized: now 1755222935558678 us; error 79 us; skew 500 ppm
I20250815 01:55:35.559545 6847 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:35.566727 6847 webserver.cc:480] Webserver started at http://127.0.106.193:34405/ using document root <none> and password file <none>
I20250815 01:55:35.567763 6847 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:35.568001 6847 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:35.576560 6847 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20250815 01:55:35.581441 6884 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:35.582651 6847 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 01:55:35.582988 6847 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
uuid: "4663eeed1fcb45768c59c9141eb245ba"
format_stamp: "Formatted at 2025-08-15 01:55:17 on dist-test-slave-h491"
I20250815 01:55:35.584913 6847 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:35.642828 6847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:35.644307 6847 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:35.644753 6847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:35.647820 6847 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:55:35.654582 6891 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 01:55:35.666226 6847 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 01:55:35.666577 6847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.001s
I20250815 01:55:35.666857 6847 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 01:55:35.671629 6847 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 01:55:35.671826 6847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250815 01:55:35.672344 6891 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Bootstrap starting.
I20250815 01:55:35.894078 6847 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:44053
I20250815 01:55:35.894240 6997 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:44053 every 8 connection(s)
I20250815 01:55:35.896863 6847 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/info.pb
I20250815 01:55:35.905771 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 6847
I20250815 01:55:35.908264 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:33735
--local_ip_for_outbound_sockets=127.0.106.194
--tserver_master_addrs=127.0.106.254:42735
--webserver_port=42559
--webserver_interface=127.0.106.194
--builtin_ntp_servers=127.0.106.212:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 01:55:35.943028 6998 heartbeater.cc:344] Connected to a master server at 127.0.106.254:42735
I20250815 01:55:35.943558 6998 heartbeater.cc:461] Registering TS with master...
I20250815 01:55:35.945057 6998 heartbeater.cc:507] Master 127.0.106.254:42735 requested a full tablet report, sending...
I20250815 01:55:35.951331 6810 ts_manager.cc:194] Registered new tserver with Master: 4663eeed1fcb45768c59c9141eb245ba (127.0.106.193:44053)
I20250815 01:55:35.959308 6810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:45703
I20250815 01:55:35.984385 6891 log.cc:826] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Log is configured to *not* fsync() on all Append() calls
W20250815 01:55:36.246587 7002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:36.247097 7002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:36.247601 7002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:36.280803 7002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:36.281663 7002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250815 01:55:36.317327 7002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:33735
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=42559
--tserver_master_addrs=127.0.106.254:42735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:36.318753 7002 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:36.320389 7002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 01:55:36.335139 7009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:55:36.964216 6998 heartbeater.cc:499] Master 127.0.106.254:42735 was elected leader, sending a full tablet report...
W20250815 01:55:37.738286 7008 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7002
W20250815 01:55:37.942564 7008 kernel_stack_watchdog.cc:198] Thread 7002 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 01:55:36.335810 7010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:37.948521 7002 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.612s user 0.571s sys 0.968s
W20250815 01:55:37.949139 7002 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.613s user 0.572s sys 0.968s
W20250815 01:55:37.950518 7012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:55:37.951963 7002 server_base.cc:1047] running on GCE node
I20250815 01:55:37.953083 7002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:37.955688 7002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:37.957183 7002 hybrid_clock.cc:648] HybridClock initialized: now 1755222937957142 us; error 39 us; skew 500 ppm
I20250815 01:55:37.958200 7002 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:37.965899 7002 webserver.cc:480] Webserver started at http://127.0.106.194:42559/ using document root <none> and password file <none>
I20250815 01:55:37.966904 7002 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:37.967151 7002 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:37.975836 7002 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.008s sys 0.001s
I20250815 01:55:37.980947 7019 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:37.982106 7002 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 01:55:37.982468 7002 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
uuid: "7072378e90f94886a6c5ef09dff30b3d"
format_stamp: "Formatted at 2025-08-15 01:55:19 on dist-test-slave-h491"
I20250815 01:55:37.984396 7002 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:38.039431 7002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:38.040887 7002 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:38.041325 7002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:38.044420 7002 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:55:38.050750 7026 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 01:55:38.057993 7002 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 01:55:38.058219 7002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.002s
I20250815 01:55:38.058529 7002 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 01:55:38.063095 7002 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 01:55:38.063345 7002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250815 01:55:38.063719 7026 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Bootstrap starting.
I20250815 01:55:38.255805 7002 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:33735
I20250815 01:55:38.256246 7132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:33735 every 8 connection(s)
I20250815 01:55:38.259503 7002 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/info.pb
I20250815 01:55:38.267068 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 7002
I20250815 01:55:38.268913 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
/tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:44785
--local_ip_for_outbound_sockets=127.0.106.195
--tserver_master_addrs=127.0.106.254:42735
--webserver_port=40377
--webserver_interface=127.0.106.195
--builtin_ntp_servers=127.0.106.212:40825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 01:55:38.314888 7133 heartbeater.cc:344] Connected to a master server at 127.0.106.254:42735
I20250815 01:55:38.315419 7133 heartbeater.cc:461] Registering TS with master...
I20250815 01:55:38.316656 7133 heartbeater.cc:507] Master 127.0.106.254:42735 requested a full tablet report, sending...
I20250815 01:55:38.320750 6810 ts_manager.cc:194] Registered new tserver with Master: 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735)
I20250815 01:55:38.323904 6810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:33503
I20250815 01:55:38.417740 7026 log.cc:826] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Log is configured to *not* fsync() on all Append() calls
W20250815 01:55:38.741242 7137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 01:55:38.741895 7137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 01:55:38.742686 7137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 01:55:38.798734 7137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 01:55:38.800055 7137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250815 01:55:38.863175 7137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:44785
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=40377
--tserver_master_addrs=127.0.106.254:42735
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 01:43:21 UTC on 5fd53c4cbb9d
build id 7573
TSAN enabled
I20250815 01:55:38.864943 7137 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 01:55:38.867081 7137 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250815 01:55:38.876286 6891 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:55:38.877259 6891 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Bootstrap complete.
I20250815 01:55:38.879230 6891 ts_tablet_manager.cc:1397] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Time spent bootstrapping tablet: real 3.207s user 3.107s sys 0.092s
W20250815 01:55:38.882179 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 01:55:38.907084 6891 raft_consensus.cc:357] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:38.922493 6891 raft_consensus.cc:738] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4663eeed1fcb45768c59c9141eb245ba, State: Initialized, Role: FOLLOWER
I20250815 01:55:38.923692 6891 consensus_queue.cc:260] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:38.931144 6891 ts_tablet_manager.cc:1428] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Time spent starting tablet: real 0.052s user 0.028s sys 0.000s
I20250815 01:55:39.328907 7133 heartbeater.cc:499] Master 127.0.106.254:42735 was elected leader, sending a full tablet report...
I20250815 01:55:40.363462 7151 raft_consensus.cc:491] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:55:40.364387 7151 raft_consensus.cc:513] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:40.386375 7151 leader_election.cc:290] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735), a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785)
W20250815 01:55:40.387373 6888 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.195:44785: connect: Connection refused (error 111)
W20250815 01:55:40.400807 6888 leader_election.cc:336] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785): Network error: Client connection negotiation failed: client connection to 127.0.106.195:44785: connect: Connection refused (error 111)
I20250815 01:55:40.413563 7088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7072378e90f94886a6c5ef09dff30b3d" is_pre_election: true
W20250815 01:55:40.429314 6886 leader_election.cc:343] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735): Illegal state: must be running to vote when last-logged opid is not known
I20250815 01:55:40.429807 6886 leader_election.cc:304] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4663eeed1fcb45768c59c9141eb245ba; no voters: 7072378e90f94886a6c5ef09dff30b3d, a4df3b043e894188b087c5b7a505dcf8
I20250815 01:55:40.430745 7151 raft_consensus.cc:2747] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250815 01:55:38.886330 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:40.287607 7143 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7137
W20250815 01:55:40.705452 7137 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.820s user 0.619s sys 1.153s
W20250815 01:55:40.707234 7137 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.822s user 0.619s sys 1.154s
W20250815 01:55:40.710232 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 01:55:40.714152 7146 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1827 milliseconds
I20250815 01:55:40.714181 7137 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 01:55:40.715665 7137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 01:55:40.718417 7137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 01:55:40.719914 7137 hybrid_clock.cc:648] HybridClock initialized: now 1755222940719881 us; error 42 us; skew 500 ppm
I20250815 01:55:40.720944 7137 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 01:55:40.729032 7137 webserver.cc:480] Webserver started at http://127.0.106.195:40377/ using document root <none> and password file <none>
I20250815 01:55:40.730412 7137 fs_manager.cc:362] Metadata directory not provided
I20250815 01:55:40.730736 7137 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 01:55:40.741869 7137 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.001s sys 0.005s
I20250815 01:55:40.748184 7161 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 01:55:40.749431 7137 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 01:55:40.749835 7137 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
uuid: "a4df3b043e894188b087c5b7a505dcf8"
format_stamp: "Formatted at 2025-08-15 01:55:21 on dist-test-slave-h491"
I20250815 01:55:40.752707 7137 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 01:55:40.814580 7137 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 01:55:40.816125 7137 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 01:55:40.816584 7137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 01:55:40.819414 7137 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 01:55:40.826579 7168 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 01:55:40.838132 7137 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 01:55:40.838501 7137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.001s
I20250815 01:55:40.838832 7137 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 01:55:40.845443 7137 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 01:55:40.845669 7137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.004s sys 0.000s
I20250815 01:55:40.846163 7168 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Bootstrap starting.
I20250815 01:55:41.067329 7137 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:44785
I20250815 01:55:41.067476 7274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:44785 every 8 connection(s)
I20250815 01:55:41.071132 7137 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/info.pb
I20250815 01:55:41.072140 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu as pid 7137
I20250815 01:55:41.130565 7275 heartbeater.cc:344] Connected to a master server at 127.0.106.254:42735
I20250815 01:55:41.131098 7275 heartbeater.cc:461] Registering TS with master...
I20250815 01:55:41.132360 7275 heartbeater.cc:507] Master 127.0.106.254:42735 requested a full tablet report, sending...
I20250815 01:55:41.136323 6810 ts_manager.cc:194] Registered new tserver with Master: a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785)
I20250815 01:55:41.139364 6810 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:39225
I20250815 01:55:41.141800 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 01:55:41.183010 7168 log.cc:826] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Log is configured to *not* fsync() on all Append() calls
I20250815 01:55:41.772184 7026 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:55:41.773254 7026 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Bootstrap complete.
I20250815 01:55:41.774600 7026 ts_tablet_manager.cc:1397] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Time spent bootstrapping tablet: real 3.711s user 3.579s sys 0.056s
I20250815 01:55:41.780038 7026 raft_consensus.cc:357] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:41.781971 7026 raft_consensus.cc:738] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7072378e90f94886a6c5ef09dff30b3d, State: Initialized, Role: FOLLOWER
I20250815 01:55:41.782888 7026 consensus_queue.cc:260] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:41.786587 7026 ts_tablet_manager.cc:1428] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20250815 01:55:42.143599 7275 heartbeater.cc:499] Master 127.0.106.254:42735 was elected leader, sending a full tablet report...
I20250815 01:55:42.400450 7289 raft_consensus.cc:491] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:55:42.400818 7289 raft_consensus.cc:513] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:42.402331 7289 leader_election.cc:290] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735), a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785)
I20250815 01:55:42.403352 7088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7072378e90f94886a6c5ef09dff30b3d" is_pre_election: true
I20250815 01:55:42.404181 7088 raft_consensus.cc:2466] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4663eeed1fcb45768c59c9141eb245ba in term 1.
I20250815 01:55:42.405893 6886 leader_election.cc:304] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4663eeed1fcb45768c59c9141eb245ba, 7072378e90f94886a6c5ef09dff30b3d; no voters:
I20250815 01:55:42.407667 7289 raft_consensus.cc:2802] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 01:55:42.408105 7289 raft_consensus.cc:491] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:55:42.408510 7289 raft_consensus.cc:3058] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:55:42.416796 7289 raft_consensus.cc:513] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:42.418685 7289 leader_election.cc:290] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 election: Requested vote from peers 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735), a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785)
I20250815 01:55:42.419566 7088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7072378e90f94886a6c5ef09dff30b3d"
I20250815 01:55:42.420132 7088 raft_consensus.cc:3058] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 1 FOLLOWER]: Advancing to term 2
I20250815 01:55:42.426786 7088 raft_consensus.cc:2466] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4663eeed1fcb45768c59c9141eb245ba in term 2.
I20250815 01:55:42.427783 6886 leader_election.cc:304] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4663eeed1fcb45768c59c9141eb245ba, 7072378e90f94886a6c5ef09dff30b3d; no voters:
I20250815 01:55:42.428402 7289 raft_consensus.cc:2802] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 2 FOLLOWER]: Leader election won for term 2
I20250815 01:55:42.430909 7289 raft_consensus.cc:695] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 2 LEADER]: Becoming Leader. State: Replica: 4663eeed1fcb45768c59c9141eb245ba, State: Running, Role: LEADER
I20250815 01:55:42.420889 7229 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "a4df3b043e894188b087c5b7a505dcf8"
I20250815 01:55:42.432215 7289 consensus_queue.cc:237] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:42.420394 7230 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c9445a6b425407eb992dd737ab6b654" candidate_uuid: "4663eeed1fcb45768c59c9141eb245ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "a4df3b043e894188b087c5b7a505dcf8" is_pre_election: true
W20250815 01:55:42.433460 6888 leader_election.cc:343] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785): Illegal state: must be running to vote when last-logged opid is not known
W20250815 01:55:42.434481 6888 leader_election.cc:343] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785): Illegal state: must be running to vote when last-logged opid is not known
I20250815 01:55:42.442958 6809 catalog_manager.cc:5582] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba reported cstate change: term changed from 0 to 2, leader changed from <none> to 4663eeed1fcb45768c59c9141eb245ba (127.0.106.193), VOTER 4663eeed1fcb45768c59c9141eb245ba (127.0.106.193) added, VOTER 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194) added, VOTER a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195) added. New cstate: current_term: 2 leader_uuid: "4663eeed1fcb45768c59c9141eb245ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } health_report { overall_health: UNKNOWN } } }
I20250815 01:55:42.966425 7088 raft_consensus.cc:1273] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 2 FOLLOWER]: Refusing update from remote peer 4663eeed1fcb45768c59c9141eb245ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250815 01:55:42.968369 7302 consensus_queue.cc:1035] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250815 01:55:43.032481 6888 consensus_peers.cc:489] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba -> Peer a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785): Couldn't send request to peer a4df3b043e894188b087c5b7a505dcf8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250815 01:55:43.062445 6953 consensus_queue.cc:237] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:43.068001 7088 raft_consensus.cc:1273] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 2 FOLLOWER]: Refusing update from remote peer 4663eeed1fcb45768c59c9141eb245ba: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250815 01:55:43.069561 7305 consensus_queue.cc:1035] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250815 01:55:43.076288 7305 raft_consensus.cc:2953] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } }
I20250815 01:55:43.078881 7088 raft_consensus.cc:2953] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } }
I20250815 01:55:43.100068 6794 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5c9445a6b425407eb992dd737ab6b654 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 01:55:43.105752 6809 catalog_manager.cc:5582] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba reported cstate change: config changed from index -1 to 207, VOTER a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195) evicted. New cstate: current_term: 2 leader_uuid: "4663eeed1fcb45768c59c9141eb245ba" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } health_report { overall_health: HEALTHY } } }
I20250815 01:55:43.137236 6953 consensus_queue.cc:237] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:43.139920 7305 raft_consensus.cc:2953] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } }
W20250815 01:55:43.142874 427 scanner-internal.cc:458] Time spent opening tablet: real 1.959s user 0.008s sys 0.001s
I20250815 01:55:43.149442 6794 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5c9445a6b425407eb992dd737ab6b654 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 01:55:43.154158 6809 catalog_manager.cc:5582] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba reported cstate change: config changed from index 207 to 208, VOTER 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194) evicted. New cstate: current_term: 2 leader_uuid: "4663eeed1fcb45768c59c9141eb245ba" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } health_report { overall_health: HEALTHY } } }
I20250815 01:55:43.177327 7210 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c9445a6b425407eb992dd737ab6b654 with delete_type TABLET_DATA_TOMBSTONED (TS a4df3b043e894188b087c5b7a505dcf8 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:32812
W20250815 01:55:43.186053 6797 catalog_manager.cc:4908] TS a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785): delete failed for tablet 5c9445a6b425407eb992dd737ab6b654 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c9445a6b425407eb992dd737ab6b654 already in progress: opening tablet
I20250815 01:55:43.192080 7068 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c9445a6b425407eb992dd737ab6b654 with delete_type TABLET_DATA_TOMBSTONED (TS 7072378e90f94886a6c5ef09dff30b3d not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:53074
I20250815 01:55:43.203785 7315 tablet_replica.cc:331] stopping tablet replica
I20250815 01:55:43.204725 7315 raft_consensus.cc:2241] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 01:55:43.205463 7315 raft_consensus.cc:2270] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 01:55:43.236888 7315 ts_tablet_manager.cc:1905] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 01:55:43.276343 7315 ts_tablet_manager.cc:1918] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250815 01:55:43.276911 7315 log.cc:1199] T 5c9445a6b425407eb992dd737ab6b654 P 7072378e90f94886a6c5ef09dff30b3d: Deleting WAL directory at /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/wal/wals/5c9445a6b425407eb992dd737ab6b654
I20250815 01:55:43.279130 6795 catalog_manager.cc:4928] TS 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735): tablet 5c9445a6b425407eb992dd737ab6b654 (table pre_rebuild [id=3f0c8ddfecad442ebd5cad60f5a8a6f0]) successfully deleted
I20250815 01:55:43.314006 7168 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 01:55:43.315282 7168 tablet_bootstrap.cc:492] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Bootstrap complete.
I20250815 01:55:43.317091 7168 ts_tablet_manager.cc:1397] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Time spent bootstrapping tablet: real 2.471s user 2.357s sys 0.092s
I20250815 01:55:43.325011 7168 raft_consensus.cc:357] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:43.328043 7168 raft_consensus.cc:738] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4df3b043e894188b087c5b7a505dcf8, State: Initialized, Role: FOLLOWER
I20250815 01:55:43.328907 7168 consensus_queue.cc:260] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } }
I20250815 01:55:43.332341 7168 ts_tablet_manager.cc:1428] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20250815 01:55:43.337337 7210 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c9445a6b425407eb992dd737ab6b654 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:32812
I20250815 01:55:43.338584 7314 tablet_replica.cc:331] stopping tablet replica
I20250815 01:55:43.339321 7314 raft_consensus.cc:2241] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250815 01:55:43.339838 7314 raft_consensus.cc:2270] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250815 01:55:43.365190 7314 ts_tablet_manager.cc:1905] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 01:55:43.383424 7314 ts_tablet_manager.cc:1918] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250815 01:55:43.383880 7314 log.cc:1199] T 5c9445a6b425407eb992dd737ab6b654 P a4df3b043e894188b087c5b7a505dcf8: Deleting WAL directory at /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/wal/wals/5c9445a6b425407eb992dd737ab6b654
I20250815 01:55:43.385619 6797 catalog_manager.cc:4928] TS a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195:44785): tablet 5c9445a6b425407eb992dd737ab6b654 (table pre_rebuild [id=3f0c8ddfecad442ebd5cad60f5a8a6f0]) successfully deleted
I20250815 01:55:43.899812 6933 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 01:55:43.916501 7210 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 01:55:43.923110 7068 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
95c8e4b59e8942aba1b04c3b909d6458 | 127.0.106.254:42735 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:40825 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
4663eeed1fcb45768c59c9141eb245ba | 127.0.106.193:44053 | HEALTHY | <none> | 1 | 0
7072378e90f94886a6c5ef09dff30b3d | 127.0.106.194:33735 | HEALTHY | <none> | 0 | 0
a4df3b043e894188b087c5b7a505dcf8 | 127.0.106.195:44785 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.193 | experimental | 127.0.106.193:44053
local_ip_for_outbound_sockets | 127.0.106.194 | experimental | 127.0.106.194:33735
local_ip_for_outbound_sockets | 127.0.106.195 | experimental | 127.0.106.195:44785
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/info.pb | hidden | 127.0.106.193:44053
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/info.pb | hidden | 127.0.106.194:33735
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/info.pb | hidden | 127.0.106.195:44785
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:40825 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 01:55:44.238190 427 log_verifier.cc:126] Checking tablet 5c9445a6b425407eb992dd737ab6b654
I20250815 01:55:44.512632 427 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 5c9445a6b425407eb992dd737ab6b654
I20250815 01:55:44.515205 6809 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46066:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250815 01:55:44.515684 6809 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46066:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250815 01:55:44.527978 6809 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 95c8e4b59e8942aba1b04c3b909d6458: Sending DeleteTablet for 1 replicas of tablet 5c9445a6b425407eb992dd737ab6b654
I20250815 01:55:44.529701 427 test_util.cc:276] Using random seed: -1340326659
I20250815 01:55:44.530544 6933 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c9445a6b425407eb992dd737ab6b654 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-15 01:55:44 UTC) from {username='slave'} at 127.0.0.1:54744
I20250815 01:55:44.533114 7347 tablet_replica.cc:331] stopping tablet replica
I20250815 01:55:44.534173 7347 raft_consensus.cc:2241] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 2 LEADER]: Raft consensus shutting down.
I20250815 01:55:44.535465 7347 raft_consensus.cc:2270] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 01:55:44.565831 6809 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44504:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250815 01:55:44.568420 6809 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 01:55:44.583855 7347 ts_tablet_manager.cc:1905] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250815 01:55:44.594659 7068 tablet_service.cc:1468] Processing CreateTablet for tablet 8c45e6451fd8479aa67c2106b7a3d8cd (DEFAULT_TABLE table=post_rebuild [id=f77a51c976c349019f51233481837519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:55:44.595579 7210 tablet_service.cc:1468] Processing CreateTablet for tablet 8c45e6451fd8479aa67c2106b7a3d8cd (DEFAULT_TABLE table=post_rebuild [id=f77a51c976c349019f51233481837519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:55:44.596052 7068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c45e6451fd8479aa67c2106b7a3d8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:44.596848 7210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c45e6451fd8479aa67c2106b7a3d8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:44.600888 7347 ts_tablet_manager.cc:1918] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250815 01:55:44.601440 7347 log.cc:1199] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Deleting WAL directory at /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/wal/wals/5c9445a6b425407eb992dd737ab6b654
I20250815 01:55:44.602567 7347 ts_tablet_manager.cc:1939] T 5c9445a6b425407eb992dd737ab6b654 P 4663eeed1fcb45768c59c9141eb245ba: Deleting consensus metadata
I20250815 01:55:44.606287 6794 catalog_manager.cc:4928] TS 4663eeed1fcb45768c59c9141eb245ba (127.0.106.193:44053): tablet 5c9445a6b425407eb992dd737ab6b654 (table pre_rebuild [id=3f0c8ddfecad442ebd5cad60f5a8a6f0]) successfully deleted
I20250815 01:55:44.610381 6933 tablet_service.cc:1468] Processing CreateTablet for tablet 8c45e6451fd8479aa67c2106b7a3d8cd (DEFAULT_TABLE table=post_rebuild [id=f77a51c976c349019f51233481837519]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 01:55:44.611948 6933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c45e6451fd8479aa67c2106b7a3d8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 01:55:44.625198 7354 tablet_bootstrap.cc:492] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d: Bootstrap starting.
I20250815 01:55:44.632637 7354 tablet_bootstrap.cc:654] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:44.642623 7355 tablet_bootstrap.cc:492] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8: Bootstrap starting.
I20250815 01:55:44.644562 7357 tablet_bootstrap.cc:492] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba: Bootstrap starting.
I20250815 01:55:44.645023 7354 tablet_bootstrap.cc:492] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d: No bootstrap required, opened a new log
I20250815 01:55:44.645545 7354 ts_tablet_manager.cc:1397] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.000s
I20250815 01:55:44.648618 7354 raft_consensus.cc:357] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:44.649550 7354 raft_consensus.cc:383] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:44.650002 7354 raft_consensus.cc:738] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7072378e90f94886a6c5ef09dff30b3d, State: Initialized, Role: FOLLOWER
I20250815 01:55:44.650894 7354 consensus_queue.cc:260] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:44.653226 7357 tablet_bootstrap.cc:654] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:44.658509 7355 tablet_bootstrap.cc:654] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8: Neither blocks nor log segments found. Creating new log.
I20250815 01:55:44.665468 7354 ts_tablet_manager.cc:1428] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d: Time spent starting tablet: real 0.020s user 0.001s sys 0.015s
I20250815 01:55:44.668056 7357 tablet_bootstrap.cc:492] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba: No bootstrap required, opened a new log
I20250815 01:55:44.668517 7357 ts_tablet_manager.cc:1397] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.007s
I20250815 01:55:44.671094 7355 tablet_bootstrap.cc:492] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8: No bootstrap required, opened a new log
I20250815 01:55:44.672194 7355 ts_tablet_manager.cc:1397] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.007s
I20250815 01:55:44.672031 7357 raft_consensus.cc:357] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:44.673395 7357 raft_consensus.cc:383] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:44.673767 7357 raft_consensus.cc:738] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4663eeed1fcb45768c59c9141eb245ba, State: Initialized, Role: FOLLOWER
I20250815 01:55:44.674639 7357 consensus_queue.cc:260] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:44.675662 7355 raft_consensus.cc:357] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:44.676513 7355 raft_consensus.cc:383] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 01:55:44.676867 7355 raft_consensus.cc:738] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4df3b043e894188b087c5b7a505dcf8, State: Initialized, Role: FOLLOWER
I20250815 01:55:44.677769 7357 ts_tablet_manager.cc:1428] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20250815 01:55:44.678686 7355 consensus_queue.cc:260] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:44.682247 7355 ts_tablet_manager.cc:1428] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
W20250815 01:55:44.684192 6999 tablet.cc:2378] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 01:55:44.785048 7134 tablet.cc:2378] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 01:55:44.865201 7280 tablet.cc:2378] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 01:55:45.052836 7362 raft_consensus.cc:491] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 01:55:45.053473 7362 raft_consensus.cc:513] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:45.056977 7362 leader_election.cc:290] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735), 4663eeed1fcb45768c59c9141eb245ba (127.0.106.193:44053)
I20250815 01:55:45.090253 6953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c45e6451fd8479aa67c2106b7a3d8cd" candidate_uuid: "a4df3b043e894188b087c5b7a505dcf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4663eeed1fcb45768c59c9141eb245ba" is_pre_election: true
I20250815 01:55:45.091363 6953 raft_consensus.cc:2466] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4df3b043e894188b087c5b7a505dcf8 in term 0.
I20250815 01:55:45.093262 7162 leader_election.cc:304] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4663eeed1fcb45768c59c9141eb245ba, a4df3b043e894188b087c5b7a505dcf8; no voters:
I20250815 01:55:45.094272 7362 raft_consensus.cc:2802] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 01:55:45.094659 7362 raft_consensus.cc:491] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 01:55:45.094960 7362 raft_consensus.cc:3058] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:45.099469 7088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c45e6451fd8479aa67c2106b7a3d8cd" candidate_uuid: "a4df3b043e894188b087c5b7a505dcf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7072378e90f94886a6c5ef09dff30b3d" is_pre_election: true
I20250815 01:55:45.100124 7088 raft_consensus.cc:2466] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4df3b043e894188b087c5b7a505dcf8 in term 0.
I20250815 01:55:45.101712 7362 raft_consensus.cc:513] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:45.104329 7088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c45e6451fd8479aa67c2106b7a3d8cd" candidate_uuid: "a4df3b043e894188b087c5b7a505dcf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7072378e90f94886a6c5ef09dff30b3d"
I20250815 01:55:45.104902 7088 raft_consensus.cc:3058] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:45.105396 6953 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c45e6451fd8479aa67c2106b7a3d8cd" candidate_uuid: "a4df3b043e894188b087c5b7a505dcf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4663eeed1fcb45768c59c9141eb245ba"
I20250815 01:55:45.105885 6953 raft_consensus.cc:3058] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba [term 0 FOLLOWER]: Advancing to term 1
I20250815 01:55:45.107827 7362 leader_election.cc:290] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [CANDIDATE]: Term 1 election: Requested vote from peers 7072378e90f94886a6c5ef09dff30b3d (127.0.106.194:33735), 4663eeed1fcb45768c59c9141eb245ba (127.0.106.193:44053)
I20250815 01:55:45.112311 7088 raft_consensus.cc:2466] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4df3b043e894188b087c5b7a505dcf8 in term 1.
I20250815 01:55:45.113734 7163 leader_election.cc:304] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7072378e90f94886a6c5ef09dff30b3d, a4df3b043e894188b087c5b7a505dcf8; no voters:
I20250815 01:55:45.115726 7362 raft_consensus.cc:2802] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 01:55:45.116626 6953 raft_consensus.cc:2466] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4df3b043e894188b087c5b7a505dcf8 in term 1.
I20250815 01:55:45.124913 7362 raft_consensus.cc:695] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [term 1 LEADER]: Becoming Leader. State: Replica: a4df3b043e894188b087c5b7a505dcf8, State: Running, Role: LEADER
I20250815 01:55:45.126107 7362 consensus_queue.cc:237] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } }
I20250815 01:55:45.141326 6809 catalog_manager.cc:5582] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4df3b043e894188b087c5b7a505dcf8 (127.0.106.195). New cstate: current_term: 1 leader_uuid: "a4df3b043e894188b087c5b7a505dcf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4df3b043e894188b087c5b7a505dcf8" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 } health_report { overall_health: UNKNOWN } } }
I20250815 01:55:45.325203 7088 raft_consensus.cc:1273] T 8c45e6451fd8479aa67c2106b7a3d8cd P 7072378e90f94886a6c5ef09dff30b3d [term 1 FOLLOWER]: Refusing update from remote peer a4df3b043e894188b087c5b7a505dcf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 01:55:45.325305 6953 raft_consensus.cc:1273] T 8c45e6451fd8479aa67c2106b7a3d8cd P 4663eeed1fcb45768c59c9141eb245ba [term 1 FOLLOWER]: Refusing update from remote peer a4df3b043e894188b087c5b7a505dcf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 01:55:45.326929 7368 consensus_queue.cc:1035] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7072378e90f94886a6c5ef09dff30b3d" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:55:45.327764 7362 consensus_queue.cc:1035] T 8c45e6451fd8479aa67c2106b7a3d8cd P a4df3b043e894188b087c5b7a505dcf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4663eeed1fcb45768c59c9141eb245ba" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 01:55:45.375844 7380 mvcc.cc:204] Tried to move back new op lower bound from 7189393184036065280 to 7189393183251795968. Current Snapshot: MvccSnapshot[applied={T|T < 7189393184036065280}]
I20250815 01:55:50.222262 7210 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 01:55:50.223038 7068 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 01:55:50.223336 6933 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
95c8e4b59e8942aba1b04c3b909d6458 | 127.0.106.254:42735 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:40825 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
4663eeed1fcb45768c59c9141eb245ba | 127.0.106.193:44053 | HEALTHY | <none> | 0 | 0
7072378e90f94886a6c5ef09dff30b3d | 127.0.106.194:33735 | HEALTHY | <none> | 0 | 0
a4df3b043e894188b087c5b7a505dcf8 | 127.0.106.195:44785 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.193 | experimental | 127.0.106.193:44053
local_ip_for_outbound_sockets | 127.0.106.194 | experimental | 127.0.106.194:33735
local_ip_for_outbound_sockets | 127.0.106.195 | experimental | 127.0.106.195:44785
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-0/data/info.pb | hidden | 127.0.106.193:44053
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-1/data/info.pb | hidden | 127.0.106.194:33735
server_dump_info_path | /tmp/dist-test-taskUhCdIZ/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755222754379616-427-0/minicluster-data/ts-2/data/info.pb | hidden | 127.0.106.195:44785
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:40825 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 01:55:50.446120 427 log_verifier.cc:126] Checking tablet 5c9445a6b425407eb992dd737ab6b654
I20250815 01:55:50.448508 427 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 5c9445a6b425407eb992dd737ab6b654
I20250815 01:55:50.448827 427 log_verifier.cc:126] Checking tablet 8c45e6451fd8479aa67c2106b7a3d8cd
I20250815 01:55:51.201735 427 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 8c45e6451fd8479aa67c2106b7a3d8cd
I20250815 01:55:51.227674 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 6847
I20250815 01:55:51.271263 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 7002
I20250815 01:55:51.307696 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 7137
I20250815 01:55:51.347731 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskUhCdIZ/build/tsan/bin/kudu with pid 6777
2025-08-15T01:55:51Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (37212 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (37212 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (196959 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST