Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250905 02:20:22.020320 15780 test_util.cc:276] Using random seed: 2060964953
W20250905 02:20:23.235553 15780 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.168s user 0.454s sys 0.710s
W20250905 02:20:23.235980 15780 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.168s user 0.454s sys 0.714s
I20250905 02:20:23.246526 15780 ts_itest-base.cc:115] Starting cluster with:
I20250905 02:20:23.246759 15780 ts_itest-base.cc:116] --------------
I20250905 02:20:23.246953 15780 ts_itest-base.cc:117] 4 tablet servers
I20250905 02:20:23.247134 15780 ts_itest-base.cc:118] 3 replicas per TS
I20250905 02:20:23.247291 15780 ts_itest-base.cc:119] --------------
2025-09-05T02:20:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T02:20:23Z Disabled control of system clock
I20250905 02:20:23.289572 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:39937
--webserver_interface=127.15.105.62
--webserver_port=0
--builtin_ntp_servers=127.15.105.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:39937 with env {}
W20250905 02:20:23.609099 15794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:23.609668 15794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:23.610118 15794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:23.642642 15794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:20:23.642982 15794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:23.643213 15794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:20:23.643406 15794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:20:23.681375 15794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:39937
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:23.682704 15794 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:23.684458 15794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:23.695271 15800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:23.697878 15801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:24.888051 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:24.891249 15802 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1188 milliseconds
I20250905 02:20:24.891480 15794 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:24.892953 15794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:24.896463 15794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:24.897954 15794 hybrid_clock.cc:648] HybridClock initialized: now 1757038824897853 us; error 121 us; skew 500 ppm
I20250905 02:20:24.898874 15794 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:24.905740 15794 webserver.cc:480] Webserver started at http://127.15.105.62:37633/ using document root <none> and password file <none>
I20250905 02:20:24.906744 15794 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:24.906951 15794 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:24.907465 15794 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:24.912274 15794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "28f5dcd1ad8840de9642cb59e2c68b5d"
format_stamp: "Formatted at 2025-09-05 02:20:24 on dist-test-slave-ctw4"
I20250905 02:20:24.913468 15794 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "28f5dcd1ad8840de9642cb59e2c68b5d"
format_stamp: "Formatted at 2025-09-05 02:20:24 on dist-test-slave-ctw4"
I20250905 02:20:24.921357 15794 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250905 02:20:24.927524 15810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:24.929025 15794 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250905 02:20:24.929421 15794 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "28f5dcd1ad8840de9642cb59e2c68b5d"
format_stamp: "Formatted at 2025-09-05 02:20:24 on dist-test-slave-ctw4"
I20250905 02:20:24.929804 15794 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:24.983901 15794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:24.985472 15794 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:24.985939 15794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:25.063375 15794 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:39937
I20250905 02:20:25.063457 15861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:39937 every 8 connection(s)
I20250905 02:20:25.066176 15794 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:20:25.072535 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 15794
I20250905 02:20:25.073186 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 02:20:25.073580 15862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:25.095849 15862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Bootstrap starting.
I20250905 02:20:25.103246 15862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:25.105791 15862 log.cc:826] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:25.112274 15862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: No bootstrap required, opened a new log
I20250905 02:20:25.131639 15862 raft_consensus.cc:357] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } }
I20250905 02:20:25.132442 15862 raft_consensus.cc:383] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:25.132694 15862 raft_consensus.cc:738] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f5dcd1ad8840de9642cb59e2c68b5d, State: Initialized, Role: FOLLOWER
I20250905 02:20:25.133453 15862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } }
I20250905 02:20:25.134006 15862 raft_consensus.cc:397] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:25.134277 15862 raft_consensus.cc:491] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:25.134581 15862 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:25.138700 15862 raft_consensus.cc:513] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } }
I20250905 02:20:25.139448 15862 leader_election.cc:304] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28f5dcd1ad8840de9642cb59e2c68b5d; no voters:
I20250905 02:20:25.141371 15862 leader_election.cc:290] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:25.142089 15867 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:25.144189 15867 raft_consensus.cc:695] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 1 LEADER]: Becoming Leader. State: Replica: 28f5dcd1ad8840de9642cb59e2c68b5d, State: Running, Role: LEADER
I20250905 02:20:25.145010 15867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } }
I20250905 02:20:25.146023 15862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:25.155395 15868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } } }
I20250905 02:20:25.155520 15869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f5dcd1ad8840de9642cb59e2c68b5d. Latest consensus state: current_term: 1 leader_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } } }
I20250905 02:20:25.156687 15868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:25.156821 15869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:25.161497 15874 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:20:25.175040 15874 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:25.193907 15874 catalog_manager.cc:1349] Generated new cluster ID: 19f7060ccc794dc0a303e2b49ac9e41f
I20250905 02:20:25.194248 15874 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:25.216626 15874 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:20:25.218605 15874 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:25.230882 15874 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Generated new TSK 0
I20250905 02:20:25.232182 15874 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:25.248708 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:0
--local_ip_for_outbound_sockets=127.15.105.1
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39937
--builtin_ntp_servers=127.15.105.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 02:20:25.579171 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:25.579689 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:25.580242 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:25.612766 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:25.613675 15886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:20:25.650449 15886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39937
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:25.652205 15886 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:25.654214 15886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:25.670006 15892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:27.072830 15891 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 15886
W20250905 02:20:27.405349 15886 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.734s user 0.672s sys 1.026s
W20250905 02:20:27.406639 15886 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.736s user 0.672s sys 1.027s
W20250905 02:20:25.671295 15893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:27.409814 15895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:27.412335 15886 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 02:20:27.412328 15894 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1735 milliseconds
I20250905 02:20:27.413584 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:27.415798 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:27.417167 15886 hybrid_clock.cc:648] HybridClock initialized: now 1757038827417117 us; error 65 us; skew 500 ppm
I20250905 02:20:27.417958 15886 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:27.425334 15886 webserver.cc:480] Webserver started at http://127.15.105.1:40773/ using document root <none> and password file <none>
I20250905 02:20:27.426332 15886 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:27.426566 15886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:27.427021 15886 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:27.431540 15886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e9000114bbfa4071b90467539127f276"
format_stamp: "Formatted at 2025-09-05 02:20:27 on dist-test-slave-ctw4"
I20250905 02:20:27.432749 15886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e9000114bbfa4071b90467539127f276"
format_stamp: "Formatted at 2025-09-05 02:20:27 on dist-test-slave-ctw4"
I20250905 02:20:27.441390 15886 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20250905 02:20:27.449617 15902 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:27.450927 15886 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.004s
I20250905 02:20:27.451292 15886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e9000114bbfa4071b90467539127f276"
format_stamp: "Formatted at 2025-09-05 02:20:27 on dist-test-slave-ctw4"
I20250905 02:20:27.451624 15886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:27.510957 15886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:27.513363 15886 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:27.513819 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:27.516923 15886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:27.521652 15886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:27.521854 15886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:27.522135 15886 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:27.522291 15886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:27.714418 15886 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:38455
I20250905 02:20:27.714622 16014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:38455 every 8 connection(s)
I20250905 02:20:27.717186 15886 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:20:27.721475 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 15886
I20250905 02:20:27.722098 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 02:20:27.736949 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:0
--local_ip_for_outbound_sockets=127.15.105.2
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39937
--builtin_ntp_servers=127.15.105.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:20:27.761958 16015 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39937
I20250905 02:20:27.762677 16015 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:27.764304 16015 heartbeater.cc:507] Master 127.15.105.62:39937 requested a full tablet report, sending...
I20250905 02:20:27.767927 15827 ts_manager.cc:194] Registered new tserver with Master: e9000114bbfa4071b90467539127f276 (127.15.105.1:38455)
I20250905 02:20:27.771453 15827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:36023
W20250905 02:20:28.080550 16019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:28.081084 16019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:28.082047 16019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:28.113696 16019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:28.114658 16019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:20:28.150621 16019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39937
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:28.151975 16019 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:28.153657 16019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:28.167207 16025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:28.776221 16015 heartbeater.cc:499] Master 127.15.105.62:39937 was elected leader, sending a full tablet report...
W20250905 02:20:28.167377 16026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:29.490736 16028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:29.492947 16027 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1320 milliseconds
W20250905 02:20:29.493830 16019 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.327s user 0.515s sys 0.803s
W20250905 02:20:29.494096 16019 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.328s user 0.515s sys 0.803s
I20250905 02:20:29.494304 16019 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:29.495314 16019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:29.497473 16019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:29.498818 16019 hybrid_clock.cc:648] HybridClock initialized: now 1757038829498792 us; error 35 us; skew 500 ppm
I20250905 02:20:29.499603 16019 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:29.507381 16019 webserver.cc:480] Webserver started at http://127.15.105.2:33727/ using document root <none> and password file <none>
I20250905 02:20:29.508389 16019 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:29.508634 16019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:29.509119 16019 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:29.513435 16019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
format_stamp: "Formatted at 2025-09-05 02:20:29 on dist-test-slave-ctw4"
I20250905 02:20:29.514492 16019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
format_stamp: "Formatted at 2025-09-05 02:20:29 on dist-test-slave-ctw4"
I20250905 02:20:29.521620 16019 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250905 02:20:29.528321 16035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:29.529327 16019 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250905 02:20:29.529623 16019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
format_stamp: "Formatted at 2025-09-05 02:20:29 on dist-test-slave-ctw4"
I20250905 02:20:29.529937 16019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:29.595009 16019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:29.596467 16019 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:29.596906 16019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:29.599717 16019 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:29.604112 16019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:29.604323 16019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:29.604566 16019 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:29.604723 16019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:29.755010 16019 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:40787
I20250905 02:20:29.755102 16147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:40787 every 8 connection(s)
I20250905 02:20:29.757699 16019 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 02:20:29.761739 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 16019
I20250905 02:20:29.762212 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 02:20:29.769338 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:0
--local_ip_for_outbound_sockets=127.15.105.3
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39937
--builtin_ntp_servers=127.15.105.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:20:29.781765 16148 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39937
I20250905 02:20:29.782308 16148 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:29.783628 16148 heartbeater.cc:507] Master 127.15.105.62:39937 requested a full tablet report, sending...
I20250905 02:20:29.786206 15827 ts_manager.cc:194] Registered new tserver with Master: 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2:40787)
I20250905 02:20:29.787542 15827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:46631
W20250905 02:20:30.093016 16152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:30.093513 16152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:30.094062 16152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:30.115492 16011 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac700 after lost signal to thread 15887
W20250905 02:20:30.116653 16011 debug-util.cc:398] Leaking SignalData structure 0x7b08000b68a0 after lost signal to thread 16014
W20250905 02:20:30.139344 16152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:30.140976 16152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:20:30.192054 16152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39937
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:30.193392 16152 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:30.195013 16152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:30.210474 16158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:30.791285 16148 heartbeater.cc:499] Master 127.15.105.62:39937 was elected leader, sending a full tablet report...
W20250905 02:20:30.214920 16159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:30.215068 16161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:31.413815 16160 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250905 02:20:31.413941 16152 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:31.418143 16152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:31.426549 16152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:31.428109 16152 hybrid_clock.cc:648] HybridClock initialized: now 1757038831428061 us; error 68 us; skew 500 ppm
I20250905 02:20:31.428953 16152 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:31.436606 16152 webserver.cc:480] Webserver started at http://127.15.105.3:37747/ using document root <none> and password file <none>
I20250905 02:20:31.437947 16152 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:31.438160 16152 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:31.438578 16152 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:31.443024 16152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6479c90677014aeca16dec729886db51"
format_stamp: "Formatted at 2025-09-05 02:20:31 on dist-test-slave-ctw4"
I20250905 02:20:31.444232 16152 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6479c90677014aeca16dec729886db51"
format_stamp: "Formatted at 2025-09-05 02:20:31 on dist-test-slave-ctw4"
I20250905 02:20:31.452479 16152 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20250905 02:20:31.458717 16168 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:31.459867 16152 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.005s
I20250905 02:20:31.460193 16152 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6479c90677014aeca16dec729886db51"
format_stamp: "Formatted at 2025-09-05 02:20:31 on dist-test-slave-ctw4"
I20250905 02:20:31.460526 16152 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:31.516805 16152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:31.518422 16152 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:31.518903 16152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:31.522748 16152 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:31.527863 16152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:31.528110 16152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:31.528350 16152 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:31.528505 16152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:31.723944 16152 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:38511
I20250905 02:20:31.724058 16280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:38511 every 8 connection(s)
I20250905 02:20:31.726603 16152 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 02:20:31.734920 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 16152
I20250905 02:20:31.735329 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 02:20:31.741063 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.4:0
--local_ip_for_outbound_sockets=127.15.105.4
--webserver_interface=127.15.105.4
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39937
--builtin_ntp_servers=127.15.105.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:20:31.759639 16281 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39937
I20250905 02:20:31.760284 16281 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:31.761796 16281 heartbeater.cc:507] Master 127.15.105.62:39937 requested a full tablet report, sending...
I20250905 02:20:31.765878 15827 ts_manager.cc:194] Registered new tserver with Master: 6479c90677014aeca16dec729886db51 (127.15.105.3:38511)
I20250905 02:20:31.768158 15827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:34939
W20250905 02:20:32.100759 16285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:32.101277 16285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:32.101764 16285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:32.133867 16285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:32.134752 16285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.4
I20250905 02:20:32.170451 16285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.105.4
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39937
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.4
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:32.171761 16285 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:32.173373 16285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:32.186514 16292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:32.773311 16281 heartbeater.cc:499] Master 127.15.105.62:39937 was elected leader, sending a full tablet report...
W20250905 02:20:32.187384 16293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:33.405900 16295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:33.409231 16294 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1216 milliseconds
I20250905 02:20:33.409319 16285 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:33.410642 16285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:33.413585 16285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:33.415063 16285 hybrid_clock.cc:648] HybridClock initialized: now 1757038833415004 us; error 75 us; skew 500 ppm
I20250905 02:20:33.415928 16285 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:33.423350 16285 webserver.cc:480] Webserver started at http://127.15.105.4:42465/ using document root <none> and password file <none>
I20250905 02:20:33.424463 16285 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:33.424769 16285 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:33.425335 16285 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:33.430531 16285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "28ec5851bba642d4a5b5fb31f46b6064"
format_stamp: "Formatted at 2025-09-05 02:20:33 on dist-test-slave-ctw4"
I20250905 02:20:33.431674 16285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "28ec5851bba642d4a5b5fb31f46b6064"
format_stamp: "Formatted at 2025-09-05 02:20:33 on dist-test-slave-ctw4"
I20250905 02:20:33.439587 16285 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20250905 02:20:33.446192 16302 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:33.447438 16285 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250905 02:20:33.447803 16285 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "28ec5851bba642d4a5b5fb31f46b6064"
format_stamp: "Formatted at 2025-09-05 02:20:33 on dist-test-slave-ctw4"
I20250905 02:20:33.448148 16285 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:33.512439 16285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:33.513969 16285 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:33.514446 16285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:33.517664 16285 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:33.523298 16285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:33.523538 16285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:33.523824 16285 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:33.524003 16285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250905 02:20:33.682521 16285 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.4:32841
I20250905 02:20:33.682652 16415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.4:32841 every 8 connection(s)
I20250905 02:20:33.685148 16285 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250905 02:20:33.695703 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 16285
I20250905 02:20:33.697237 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250905 02:20:33.718225 16416 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39937
I20250905 02:20:33.718675 16416 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:33.720269 16416 heartbeater.cc:507] Master 127.15.105.62:39937 requested a full tablet report, sending...
I20250905 02:20:33.725564 15827 ts_manager.cc:194] Registered new tserver with Master: 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841)
I20250905 02:20:33.727715 15827 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.4:60359
I20250905 02:20:33.734898 15780 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250905 02:20:33.783768 15827 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36590:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250905 02:20:33.875447 15950 tablet_service.cc:1468] Processing CreateTablet for tablet 52a66cc2002b4a88bada19091b3868c8 (DEFAULT_TABLE table=TestTable [id=57829e22a3ad485797833ef97f552e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:20:33.879824 15950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a66cc2002b4a88bada19091b3868c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:33.882790 16351 tablet_service.cc:1468] Processing CreateTablet for tablet 52a66cc2002b4a88bada19091b3868c8 (DEFAULT_TABLE table=TestTable [id=57829e22a3ad485797833ef97f552e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:20:33.883689 16083 tablet_service.cc:1468] Processing CreateTablet for tablet 52a66cc2002b4a88bada19091b3868c8 (DEFAULT_TABLE table=TestTable [id=57829e22a3ad485797833ef97f552e5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:20:33.885493 16351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a66cc2002b4a88bada19091b3868c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:33.885577 16083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a66cc2002b4a88bada19091b3868c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:33.913832 16436 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6: Bootstrap starting.
I20250905 02:20:33.914393 16435 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: Bootstrap starting.
I20250905 02:20:33.919319 16437 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064: Bootstrap starting.
I20250905 02:20:33.922633 16436 tablet_bootstrap.cc:654] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:33.923588 16435 tablet_bootstrap.cc:654] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:33.925074 16436 log.cc:826] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:33.926030 16435 log.cc:826] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:33.929896 16437 tablet_bootstrap.cc:654] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:33.931378 16436 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6: No bootstrap required, opened a new log
I20250905 02:20:33.931885 16436 ts_tablet_manager.cc:1397] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.013s
I20250905 02:20:33.933084 16437 log.cc:826] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:33.933419 16435 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: No bootstrap required, opened a new log
I20250905 02:20:33.934002 16435 ts_tablet_manager.cc:1397] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.008s
I20250905 02:20:33.941428 16437 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064: No bootstrap required, opened a new log
I20250905 02:20:33.942144 16437 ts_tablet_manager.cc:1397] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.011s
I20250905 02:20:33.953747 16436 raft_consensus.cc:357] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:33.954466 16436 raft_consensus.cc:383] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:33.954730 16436 raft_consensus.cc:738] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, State: Initialized, Role: FOLLOWER
I20250905 02:20:33.955518 16436 consensus_queue.cc:260] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:33.961511 16436 ts_tablet_manager.cc:1428] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6: Time spent starting tablet: real 0.029s user 0.025s sys 0.003s
I20250905 02:20:33.964182 16435 raft_consensus.cc:357] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:33.965440 16435 raft_consensus.cc:383] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:33.965673 16435 raft_consensus.cc:738] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9000114bbfa4071b90467539127f276, State: Initialized, Role: FOLLOWER
I20250905 02:20:33.966426 16435 consensus_queue.cc:260] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:33.969966 16437 raft_consensus.cc:357] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:33.970865 16437 raft_consensus.cc:383] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:33.971171 16437 raft_consensus.cc:738] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ec5851bba642d4a5b5fb31f46b6064, State: Initialized, Role: FOLLOWER
I20250905 02:20:33.971525 16435 ts_tablet_manager.cc:1428] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: Time spent starting tablet: real 0.037s user 0.031s sys 0.005s
I20250905 02:20:33.972054 16437 consensus_queue.cc:260] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:33.974977 16416 heartbeater.cc:499] Master 127.15.105.62:39937 was elected leader, sending a full tablet report...
I20250905 02:20:33.977697 16437 ts_tablet_manager.cc:1428] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064: Time spent starting tablet: real 0.035s user 0.032s sys 0.001s
W20250905 02:20:33.991384 16016 tablet.cc:2378] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:20:34.017709 16443 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 02:20:34.018016 16149 tablet.cc:2378] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:20:34.018254 16443 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:34.021145 16443 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9000114bbfa4071b90467539127f276 (127.15.105.1:38455), 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2:40787)
I20250905 02:20:34.029753 15970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "28ec5851bba642d4a5b5fb31f46b6064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9000114bbfa4071b90467539127f276" is_pre_election: true
I20250905 02:20:34.030441 15970 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ec5851bba642d4a5b5fb31f46b6064 in term 0.
I20250905 02:20:34.031605 16306 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28ec5851bba642d4a5b5fb31f46b6064, e9000114bbfa4071b90467539127f276; no voters:
I20250905 02:20:34.032928 16443 raft_consensus.cc:2802] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:34.032835 16103 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "28ec5851bba642d4a5b5fb31f46b6064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" is_pre_election: true
I20250905 02:20:34.033197 16443 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:34.033457 16443 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:34.033517 16103 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ec5851bba642d4a5b5fb31f46b6064 in term 0.
I20250905 02:20:34.038218 16443 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:34.039700 16443 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [CANDIDATE]: Term 1 election: Requested vote from peers e9000114bbfa4071b90467539127f276 (127.15.105.1:38455), 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2:40787)
I20250905 02:20:34.040459 15970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "28ec5851bba642d4a5b5fb31f46b6064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9000114bbfa4071b90467539127f276"
I20250905 02:20:34.040594 16103 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "28ec5851bba642d4a5b5fb31f46b6064" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
I20250905 02:20:34.040992 15970 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:34.041082 16103 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:34.048202 15970 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ec5851bba642d4a5b5fb31f46b6064 in term 1.
I20250905 02:20:34.048202 16103 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ec5851bba642d4a5b5fb31f46b6064 in term 1.
I20250905 02:20:34.049654 16305 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, 28ec5851bba642d4a5b5fb31f46b6064; no voters:
I20250905 02:20:34.050329 16443 raft_consensus.cc:2802] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:34.053310 16443 raft_consensus.cc:695] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [term 1 LEADER]: Becoming Leader. State: Replica: 28ec5851bba642d4a5b5fb31f46b6064, State: Running, Role: LEADER
I20250905 02:20:34.054244 16443 consensus_queue.cc:237] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:34.063928 15825 catalog_manager.cc:5582] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4). New cstate: current_term: 1 leader_uuid: "28ec5851bba642d4a5b5fb31f46b6064" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:34.090494 15780 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250905 02:20:34.096313 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e9000114bbfa4071b90467539127f276 to finish bootstrapping
I20250905 02:20:34.114162 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 27b3ab7b0f6d4d41a04a8eab6bbef9d6 to finish bootstrapping
I20250905 02:20:34.126111 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 28ec5851bba642d4a5b5fb31f46b6064 to finish bootstrapping
I20250905 02:20:34.138058 15780 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250905 02:20:34.141193 15780 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "52a66cc2002b4a88bada19091b3868c8"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "e9000114bbfa4071b90467539127f276"
rpc_addresses {
host: "127.15.105.1"
port: 38455
}
}
ts_infos {
permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064"
rpc_addresses {
host: "127.15.105.4"
port: 32841
}
}
ts_infos {
permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
rpc_addresses {
host: "127.15.105.2"
port: 40787
}
}
W20250905 02:20:34.191036 16417 tablet.cc:2378] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:20:34.538942 16443 consensus_queue.cc:1035] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:34.557782 16445 consensus_queue.cc:1035] T 52a66cc2002b4a88bada19091b3868c8 P 28ec5851bba642d4a5b5fb31f46b6064 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:34.563763 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 16285
W20250905 02:20:34.593634 15904 connection.cc:537] server connection from 127.15.105.4:36181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 02:20:34.594241 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 15794
I20250905 02:20:34.625172 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:39937
--webserver_interface=127.15.105.62
--webserver_port=37633
--builtin_ntp_servers=127.15.105.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:39937 with env {}
W20250905 02:20:34.786916 16281 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:39937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:39937: connect: Connection refused (error 111)
W20250905 02:20:34.966655 16457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:34.967268 16457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:34.967707 16457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:35.000914 16457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:20:35.001214 16457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:35.001420 16457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:20:35.001619 16457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:20:35.051710 16457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:39937
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=37633
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:35.053063 16457 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:35.054736 16457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:35.065708 16464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:35.582904 16148 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:39937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:39937: connect: Connection refused (error 111)
W20250905 02:20:35.592550 16015 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:39937 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:39937: connect: Connection refused (error 111)
I20250905 02:20:36.056821 16474 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28ec5851bba642d4a5b5fb31f46b6064)
I20250905 02:20:36.057519 16474 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:36.061475 16474 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9000114bbfa4071b90467539127f276 (127.15.105.1:38455), 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841)
W20250905 02:20:36.063982 16036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111)
I20250905 02:20:36.072808 16476 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 28ec5851bba642d4a5b5fb31f46b6064)
I20250905 02:20:36.075510 16476 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:36.084208 16476 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841), 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2:40787)
W20250905 02:20:36.084520 15903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111)
W20250905 02:20:36.095141 16036 leader_election.cc:336] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841): Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111)
W20250905 02:20:36.100091 15903 leader_election.cc:336] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841): Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111)
I20250905 02:20:36.119372 15970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e9000114bbfa4071b90467539127f276" is_pre_election: true
I20250905 02:20:36.120327 15970 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b3ab7b0f6d4d41a04a8eab6bbef9d6 in term 1.
I20250905 02:20:36.121382 16103 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "e9000114bbfa4071b90467539127f276" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" is_pre_election: true
I20250905 02:20:36.121938 16103 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9000114bbfa4071b90467539127f276 in term 1.
I20250905 02:20:36.122362 16039 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, e9000114bbfa4071b90467539127f276; no voters: 28ec5851bba642d4a5b5fb31f46b6064
I20250905 02:20:36.123391 15905 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, e9000114bbfa4071b90467539127f276; no voters: 28ec5851bba642d4a5b5fb31f46b6064
I20250905 02:20:36.123924 16474 raft_consensus.cc:2802] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:20:36.124207 16476 raft_consensus.cc:2802] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:20:36.124275 16474 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28ec5851bba642d4a5b5fb31f46b6064)
I20250905 02:20:36.124517 16476 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 28ec5851bba642d4a5b5fb31f46b6064)
I20250905 02:20:36.124599 16474 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:36.124828 16476 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:36.131628 16474 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:36.131808 16476 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:36.134413 15970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e9000114bbfa4071b90467539127f276"
I20250905 02:20:36.135177 15970 raft_consensus.cc:2391] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 27b3ab7b0f6d4d41a04a8eab6bbef9d6 in current term 2: Already voted for candidate e9000114bbfa4071b90467539127f276 in this term.
W20250905 02:20:36.137560 15903 leader_election.cc:336] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841): Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111)
I20250905 02:20:36.138863 16474 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 2 election: Requested vote from peers e9000114bbfa4071b90467539127f276 (127.15.105.1:38455), 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841)
I20250905 02:20:36.139533 16103 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "e9000114bbfa4071b90467539127f276" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
I20250905 02:20:36.140298 16103 raft_consensus.cc:2391] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e9000114bbfa4071b90467539127f276 in current term 2: Already voted for candidate 27b3ab7b0f6d4d41a04a8eab6bbef9d6 in this term.
W20250905 02:20:36.141031 16036 leader_election.cc:336] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841): Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111)
I20250905 02:20:36.141407 15905 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9000114bbfa4071b90467539127f276; no voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, 28ec5851bba642d4a5b5fb31f46b6064
I20250905 02:20:36.141424 16036 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6; no voters: 28ec5851bba642d4a5b5fb31f46b6064, e9000114bbfa4071b90467539127f276
I20250905 02:20:36.145798 16474 raft_consensus.cc:2747] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 02:20:36.146859 16476 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 2 election: Requested vote from peers 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841), 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2:40787)
I20250905 02:20:36.148375 16476 raft_consensus.cc:2747] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250905 02:20:35.066260 16465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:36.375205 16467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:36.377570 16466 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1305 milliseconds
I20250905 02:20:36.377669 16457 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:36.379040 16457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:36.382294 16457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:36.383764 16457 hybrid_clock.cc:648] HybridClock initialized: now 1757038836383720 us; error 56 us; skew 500 ppm
I20250905 02:20:36.384598 16457 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:36.391391 16457 webserver.cc:480] Webserver started at http://127.15.105.62:37633/ using document root <none> and password file <none>
I20250905 02:20:36.392350 16457 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:36.392561 16457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:36.400378 16457 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
I20250905 02:20:36.404760 16487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:36.405774 16457 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250905 02:20:36.406095 16457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "28f5dcd1ad8840de9642cb59e2c68b5d"
format_stamp: "Formatted at 2025-09-05 02:20:24 on dist-test-slave-ctw4"
I20250905 02:20:36.408075 16457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:36.468448 16457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:36.469902 16457 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:36.470326 16457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:36.542490 16457 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:39937
I20250905 02:20:36.542568 16538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:39937 every 8 connection(s)
I20250905 02:20:36.545303 16457 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:20:36.546169 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 16457
I20250905 02:20:36.546689 15780 kudu-admin-test.cc:735] Forcing unsafe config change on tserver e9000114bbfa4071b90467539127f276
I20250905 02:20:36.558895 16539 sys_catalog.cc:263] Verifying existing consensus state
I20250905 02:20:36.563520 16539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Bootstrap starting.
I20250905 02:20:36.609863 16148 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39937
I20250905 02:20:36.613308 16539 log.cc:826] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:36.620498 16015 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39937
I20250905 02:20:36.638482 16539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:20:36.639256 16539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Bootstrap complete.
I20250905 02:20:36.658149 16539 raft_consensus.cc:357] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } }
I20250905 02:20:36.660171 16539 raft_consensus.cc:738] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f5dcd1ad8840de9642cb59e2c68b5d, State: Initialized, Role: FOLLOWER
I20250905 02:20:36.661007 16539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } }
I20250905 02:20:36.661522 16539 raft_consensus.cc:397] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:36.661795 16539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:36.662081 16539 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:36.667506 16539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } }
I20250905 02:20:36.668293 16539 leader_election.cc:304] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28f5dcd1ad8840de9642cb59e2c68b5d; no voters:
I20250905 02:20:36.670398 16539 leader_election.cc:290] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 02:20:36.670782 16545 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:20:36.674930 16545 raft_consensus.cc:695] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [term 2 LEADER]: Becoming Leader. State: Replica: 28f5dcd1ad8840de9642cb59e2c68b5d, State: Running, Role: LEADER
I20250905 02:20:36.675827 16539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:36.675819 16545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } }
I20250905 02:20:36.686352 16546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } } }
I20250905 02:20:36.687036 16546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:36.687698 16547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f5dcd1ad8840de9642cb59e2c68b5d. Latest consensus state: current_term: 2 leader_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f5dcd1ad8840de9642cb59e2c68b5d" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39937 } } }
I20250905 02:20:36.688498 16547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:36.694685 16552 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:20:36.706924 16552 catalog_manager.cc:671] Loaded metadata for table TestTable [id=57829e22a3ad485797833ef97f552e5b]
I20250905 02:20:36.714301 16552 tablet_loader.cc:96] loaded metadata for tablet 52a66cc2002b4a88bada19091b3868c8 (table TestTable [id=57829e22a3ad485797833ef97f552e5b])
I20250905 02:20:36.716197 16552 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:36.721261 16552 catalog_manager.cc:1261] Loaded cluster ID: 19f7060ccc794dc0a303e2b49ac9e41f
I20250905 02:20:36.721652 16552 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:36.730970 16552 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:36.736295 16552 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 28f5dcd1ad8840de9642cb59e2c68b5d: Loaded TSK: 0
I20250905 02:20:36.738137 16552 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:36.807466 16281 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39937
I20250905 02:20:36.813177 16504 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6479c90677014aeca16dec729886db51" instance_seqno: 1757038831678352) as {username='slave'} at 127.15.105.3:50681; Asking this server to re-register.
I20250905 02:20:36.814530 16281 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:36.815166 16281 heartbeater.cc:507] Master 127.15.105.62:39937 requested a full tablet report, sending...
I20250905 02:20:36.817616 16504 ts_manager.cc:194] Registered new tserver with Master: 6479c90677014aeca16dec729886db51 (127.15.105.3:38511)
W20250905 02:20:36.898790 16541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:36.899365 16541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:36.929940 16541 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250905 02:20:37.618259 16504 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" instance_seqno: 1757038829719804) as {username='slave'} at 127.15.105.2:59893; Asking this server to re-register.
I20250905 02:20:37.620452 16148 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.621560 16148 heartbeater.cc:507] Master 127.15.105.62:39937 requested a full tablet report, sending...
I20250905 02:20:37.625496 16504 ts_manager.cc:194] Registered new tserver with Master: 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2:40787)
I20250905 02:20:37.627594 16503 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e9000114bbfa4071b90467539127f276" instance_seqno: 1757038827670745) as {username='slave'} at 127.15.105.1:44663; Asking this server to re-register.
I20250905 02:20:37.632632 16015 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.633553 16015 heartbeater.cc:507] Master 127.15.105.62:39937 requested a full tablet report, sending...
I20250905 02:20:37.638710 16503 ts_manager.cc:194] Registered new tserver with Master: e9000114bbfa4071b90467539127f276 (127.15.105.1:38455)
I20250905 02:20:37.711081 16573 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:37.711992 16573 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:37.715623 15970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e9000114bbfa4071b90467539127f276" is_pre_election: true
I20250905 02:20:37.716483 15970 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27b3ab7b0f6d4d41a04a8eab6bbef9d6 in term 2.
I20250905 02:20:37.718360 16039 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, e9000114bbfa4071b90467539127f276; no voters:
I20250905 02:20:37.721307 16574 raft_consensus.cc:2802] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 02:20:37.723290 16574 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:37.724388 16574 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:20:37.727911 16573 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9000114bbfa4071b90467539127f276 (127.15.105.1:38455), 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841)
I20250905 02:20:37.734092 16574 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:37.738188 15970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e9000114bbfa4071b90467539127f276"
I20250905 02:20:37.738827 15970 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:20:37.747992 15970 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27b3ab7b0f6d4d41a04a8eab6bbef9d6 in term 3.
I20250905 02:20:37.749087 16574 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 3 election: Requested vote from peers e9000114bbfa4071b90467539127f276 (127.15.105.1:38455), 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841)
I20250905 02:20:37.750715 16039 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, e9000114bbfa4071b90467539127f276; no voters:
I20250905 02:20:37.751874 16574 raft_consensus.cc:2802] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:20:37.753417 16574 raft_consensus.cc:695] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 3 LEADER]: Becoming Leader. State: Replica: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, State: Running, Role: LEADER
I20250905 02:20:37.754513 16574 consensus_queue.cc:237] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
W20250905 02:20:37.765851 16036 leader_election.cc:336] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841): Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111)
W20250905 02:20:37.768460 16036 leader_election.cc:336] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841): Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111)
I20250905 02:20:37.778852 16504 catalog_manager.cc:5582] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 reported cstate change: term changed from 1 to 3, leader changed from 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4) to 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2). New cstate: current_term: 3 leader_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:38.213594 15970 raft_consensus.cc:1273] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 3 FOLLOWER]: Refusing update from remote peer 27b3ab7b0f6d4d41a04a8eab6bbef9d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 02:20:38.215313 16574 consensus_queue.cc:1035] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250905 02:20:38.270401 16036 consensus_peers.cc:489] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 -> Peer 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4:32841): Couldn't send request to peer 28ec5851bba642d4a5b5fb31f46b6064. Status: Network error: Client connection negotiation failed: client connection to 127.15.105.4:32841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250905 02:20:38.342868 16541 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.372s user 0.584s sys 0.745s
W20250905 02:20:38.343308 16541 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.373s user 0.586s sys 0.746s
I20250905 02:20:38.413992 15970 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "e9000114bbfa4071b90467539127f276"
tablet_id: "52a66cc2002b4a88bada19091b3868c8"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
}
peers {
permanent_uuid: "e9000114bbfa4071b90467539127f276"
}
}
from {username='slave'} at 127.0.0.1:35424
W20250905 02:20:38.415310 15970 raft_consensus.cc:2216] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 3 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } unsafe_config_change: true
I20250905 02:20:38.416446 15970 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 3 FOLLOWER]: Advancing to term 4
I20250905 02:20:38.651571 15970 raft_consensus.cc:1238] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 4 FOLLOWER]: Rejecting Update request from peer 27b3ab7b0f6d4d41a04a8eab6bbef9d6 for earlier term 3. Current term is 4. Ops: []
I20250905 02:20:38.653126 16586 consensus_queue.cc:1046] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 02:20:38.653946 16574 raft_consensus.cc:3053] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 3 LEADER]: Stepping down as leader of term 3
I20250905 02:20:38.654325 16574 raft_consensus.cc:738] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, State: Running, Role: LEADER
I20250905 02:20:38.655083 16574 consensus_queue.cc:260] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } }
I20250905 02:20:38.656033 16574 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 3 FOLLOWER]: Advancing to term 4
I20250905 02:20:39.923722 16598 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250905 02:20:39.924114 16598 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } unsafe_config_change: true
I20250905 02:20:39.925135 16598 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2:40787)
I20250905 02:20:39.926293 16103 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "e9000114bbfa4071b90467539127f276" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" is_pre_election: true
I20250905 02:20:39.926819 16103 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9000114bbfa4071b90467539127f276 in term 4.
I20250905 02:20:39.927726 15905 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, e9000114bbfa4071b90467539127f276; no voters:
I20250905 02:20:39.928288 16598 raft_consensus.cc:2802] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250905 02:20:39.928503 16598 raft_consensus.cc:491] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 4 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250905 02:20:39.928751 16598 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 4 FOLLOWER]: Advancing to term 5
I20250905 02:20:39.936192 16598 raft_consensus.cc:513] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } unsafe_config_change: true
I20250905 02:20:39.937150 16598 leader_election.cc:290] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 5 election: Requested vote from peers 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2:40787)
I20250905 02:20:39.938104 16103 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52a66cc2002b4a88bada19091b3868c8" candidate_uuid: "e9000114bbfa4071b90467539127f276" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
I20250905 02:20:39.938488 16103 raft_consensus.cc:3058] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 4 FOLLOWER]: Advancing to term 5
I20250905 02:20:40.040109 16103 raft_consensus.cc:2466] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9000114bbfa4071b90467539127f276 in term 5.
I20250905 02:20:40.041042 15905 leader_election.cc:304] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 27b3ab7b0f6d4d41a04a8eab6bbef9d6, e9000114bbfa4071b90467539127f276; no voters:
I20250905 02:20:40.041633 16598 raft_consensus.cc:2802] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 5 FOLLOWER]: Leader election won for term 5
I20250905 02:20:40.042387 16598 raft_consensus.cc:695] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 5 LEADER]: Becoming Leader. State: Replica: e9000114bbfa4071b90467539127f276, State: Running, Role: LEADER
I20250905 02:20:40.043169 16598 consensus_queue.cc:237] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 0, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } unsafe_config_change: true
I20250905 02:20:40.049613 16504 catalog_manager.cc:5582] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 reported cstate change: term changed from 3 to 5, leader changed from 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2) to e9000114bbfa4071b90467539127f276 (127.15.105.1), now has a pending config: VOTER e9000114bbfa4071b90467539127f276 (127.15.105.1), VOTER 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2). New cstate: current_term: 5 leader_uuid: "e9000114bbfa4071b90467539127f276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28ec5851bba642d4a5b5fb31f46b6064" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 32841 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } unsafe_config_change: true }
I20250905 02:20:40.638370 16103 raft_consensus.cc:1273] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 5 FOLLOWER]: Refusing update from remote peer e9000114bbfa4071b90467539127f276: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250905 02:20:40.639562 16606 consensus_queue.cc:1035] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:20:40.647735 16609 raft_consensus.cc:2953] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 5 LEADER]: Committing config change with OpId 4.3: config changed from index -1 to 3, VOTER 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } unsafe_config_change: true }
I20250905 02:20:40.649083 16103 raft_consensus.cc:2953] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 5 FOLLOWER]: Committing config change with OpId 4.3: config changed from index -1 to 3, VOTER 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } unsafe_config_change: true }
I20250905 02:20:40.665783 16504 catalog_manager.cc:5582] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 reported cstate change: config changed from index -1 to 3, VOTER 28ec5851bba642d4a5b5fb31f46b6064 (127.15.105.4) evicted, no longer has a pending config: VOTER e9000114bbfa4071b90467539127f276 (127.15.105.1), VOTER 27b3ab7b0f6d4d41a04a8eab6bbef9d6 (127.15.105.2). New cstate: current_term: 5 leader_uuid: "e9000114bbfa4071b90467539127f276" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } unsafe_config_change: true }
W20250905 02:20:40.674748 16504 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 52a66cc2002b4a88bada19091b3868c8 on TS 28ec5851bba642d4a5b5fb31f46b6064: Not found: failed to reset TS proxy: Could not find TS for UUID 28ec5851bba642d4a5b5fb31f46b6064
I20250905 02:20:40.692299 15970 consensus_queue.cc:237] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 0, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: true } } unsafe_config_change: true
I20250905 02:20:40.697347 16103 raft_consensus.cc:1273] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 5 FOLLOWER]: Refusing update from remote peer e9000114bbfa4071b90467539127f276: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250905 02:20:40.699386 16606 consensus_queue.cc:1035] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 02:20:40.705289 16612 raft_consensus.cc:2953] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index 3 to 5, NON_VOTER 6479c90677014aeca16dec729886db51 (127.15.105.3) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: true } } unsafe_config_change: true }
I20250905 02:20:40.706763 16103 raft_consensus.cc:2953] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 3 to 5, NON_VOTER 6479c90677014aeca16dec729886db51 (127.15.105.3) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: true } } unsafe_config_change: true }
W20250905 02:20:40.710749 15906 consensus_peers.cc:489] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 -> Peer 6479c90677014aeca16dec729886db51 (127.15.105.3:38511): Couldn't send request to peer 6479c90677014aeca16dec729886db51. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52a66cc2002b4a88bada19091b3868c8. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:40.715322 16491 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 52a66cc2002b4a88bada19091b3868c8 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250905 02:20:40.718812 16504 catalog_manager.cc:5582] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 reported cstate change: config changed from index 3 to 5, NON_VOTER 6479c90677014aeca16dec729886db51 (127.15.105.3) added. New cstate: current_term: 5 leader_uuid: "e9000114bbfa4071b90467539127f276" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: true } } unsafe_config_change: true }
W20250905 02:20:40.727957 16489 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 52a66cc2002b4a88bada19091b3868c8 on TS 28ec5851bba642d4a5b5fb31f46b6064 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 28ec5851bba642d4a5b5fb31f46b6064
I20250905 02:20:41.090227 16620 ts_tablet_manager.cc:927] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: Initiating tablet copy from peer e9000114bbfa4071b90467539127f276 (127.15.105.1:38455)
I20250905 02:20:41.093019 16620 tablet_copy_client.cc:323] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: tablet copy: Beginning tablet copy session from remote peer at address 127.15.105.1:38455
I20250905 02:20:41.103581 15990 tablet_copy_service.cc:140] P e9000114bbfa4071b90467539127f276: Received BeginTabletCopySession request for tablet 52a66cc2002b4a88bada19091b3868c8 from peer 6479c90677014aeca16dec729886db51 ({username='slave'} at 127.15.105.3:38349)
I20250905 02:20:41.104079 15990 tablet_copy_service.cc:161] P e9000114bbfa4071b90467539127f276: Beginning new tablet copy session on tablet 52a66cc2002b4a88bada19091b3868c8 from peer 6479c90677014aeca16dec729886db51 at {username='slave'} at 127.15.105.3:38349: session id = 6479c90677014aeca16dec729886db51-52a66cc2002b4a88bada19091b3868c8
I20250905 02:20:41.109020 15990 tablet_copy_source_session.cc:215] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:20:41.115478 16620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a66cc2002b4a88bada19091b3868c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:41.135447 16620 tablet_copy_client.cc:806] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: tablet copy: Starting download of 0 data blocks...
I20250905 02:20:41.136097 16620 tablet_copy_client.cc:670] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: tablet copy: Starting download of 1 WAL segments...
I20250905 02:20:41.139575 16620 tablet_copy_client.cc:538] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:20:41.145180 16620 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: Bootstrap starting.
I20250905 02:20:41.157104 16620 log.cc:826] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:41.167254 16620 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:20:41.167932 16620 tablet_bootstrap.cc:492] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: Bootstrap complete.
I20250905 02:20:41.168422 16620 ts_tablet_manager.cc:1397] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: Time spent bootstrapping tablet: real 0.024s user 0.023s sys 0.000s
I20250905 02:20:41.184684 16620 raft_consensus.cc:357] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: true } } unsafe_config_change: true
I20250905 02:20:41.185526 16620 raft_consensus.cc:738] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 6479c90677014aeca16dec729886db51, State: Initialized, Role: LEARNER
I20250905 02:20:41.186074 16620 consensus_queue.cc:260] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: true } } unsafe_config_change: true
I20250905 02:20:41.189723 16620 ts_tablet_manager.cc:1428] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51: Time spent starting tablet: real 0.021s user 0.009s sys 0.011s
I20250905 02:20:41.191372 15990 tablet_copy_service.cc:342] P e9000114bbfa4071b90467539127f276: Request end of tablet copy session 6479c90677014aeca16dec729886db51-52a66cc2002b4a88bada19091b3868c8 received from {username='slave'} at 127.15.105.3:38349
I20250905 02:20:41.191843 15990 tablet_copy_service.cc:434] P e9000114bbfa4071b90467539127f276: ending tablet copy session 6479c90677014aeca16dec729886db51-52a66cc2002b4a88bada19091b3868c8 on tablet 52a66cc2002b4a88bada19091b3868c8 with peer 6479c90677014aeca16dec729886db51
W20250905 02:20:41.259642 16535 debug-util.cc:398] Leaking SignalData structure 0x7b08000aca00 after lost signal to thread 16459
W20250905 02:20:41.260322 16535 debug-util.cc:398] Leaking SignalData structure 0x7b080007c860 after lost signal to thread 16538
I20250905 02:20:41.522019 16236 raft_consensus.cc:1215] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
W20250905 02:20:41.894878 16489 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 52a66cc2002b4a88bada19091b3868c8 on TS 28ec5851bba642d4a5b5fb31f46b6064 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 28ec5851bba642d4a5b5fb31f46b6064
I20250905 02:20:41.958420 16626 raft_consensus.cc:1062] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276: attempting to promote NON_VOTER 6479c90677014aeca16dec729886db51 to VOTER
I20250905 02:20:41.959908 16626 consensus_queue.cc:237] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 0, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: false } } unsafe_config_change: true
I20250905 02:20:41.964589 16236 raft_consensus.cc:1273] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51 [term 5 LEARNER]: Refusing update from remote peer e9000114bbfa4071b90467539127f276: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 02:20:41.964773 16103 raft_consensus.cc:1273] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 5 FOLLOWER]: Refusing update from remote peer e9000114bbfa4071b90467539127f276: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250905 02:20:41.965678 16626 consensus_queue.cc:1035] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 02:20:41.966401 16606 consensus_queue.cc:1035] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 02:20:41.974933 16626 raft_consensus.cc:2953] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 6479c90677014aeca16dec729886db51 (127.15.105.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: false } } unsafe_config_change: true }
I20250905 02:20:41.976655 16103 raft_consensus.cc:2953] T 52a66cc2002b4a88bada19091b3868c8 P 27b3ab7b0f6d4d41a04a8eab6bbef9d6 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 6479c90677014aeca16dec729886db51 (127.15.105.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: false } } unsafe_config_change: true }
I20250905 02:20:41.978626 16235 raft_consensus.cc:2953] T 52a66cc2002b4a88bada19091b3868c8 P 6479c90677014aeca16dec729886db51 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 6479c90677014aeca16dec729886db51 (127.15.105.3) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: false } } unsafe_config_change: true }
I20250905 02:20:41.991365 16504 catalog_manager.cc:5582] T 52a66cc2002b4a88bada19091b3868c8 P e9000114bbfa4071b90467539127f276 reported cstate change: config changed from index 5 to 6, 6479c90677014aeca16dec729886db51 (127.15.105.3) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "e9000114bbfa4071b90467539127f276" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9000114bbfa4071b90467539127f276" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 38455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6479c90677014aeca16dec729886db51" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38511 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250905 02:20:42.008157 15780 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250905 02:20:42.027201 15780 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "52a66cc2002b4a88bada19091b3868c8"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "e9000114bbfa4071b90467539127f276"
rpc_addresses {
host: "127.15.105.1"
port: 38455
}
}
ts_infos {
permanent_uuid: "27b3ab7b0f6d4d41a04a8eab6bbef9d6"
rpc_addresses {
host: "127.15.105.2"
port: 40787
}
}
ts_infos {
permanent_uuid: "6479c90677014aeca16dec729886db51"
rpc_addresses {
host: "127.15.105.3"
port: 38511
}
}
I20250905 02:20:42.029349 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 15886
I20250905 02:20:42.060837 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 16019
I20250905 02:20:42.088407 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 16152
I20250905 02:20:42.123378 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 16457
2025-09-05T02:20:42Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (20176 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250905 02:20:42.192497 15780 test_util.cc:276] Using random seed: 2081137242
I20250905 02:20:42.198385 15780 ts_itest-base.cc:115] Starting cluster with:
I20250905 02:20:42.198542 15780 ts_itest-base.cc:116] --------------
I20250905 02:20:42.198658 15780 ts_itest-base.cc:117] 3 tablet servers
I20250905 02:20:42.198763 15780 ts_itest-base.cc:118] 3 replicas per TS
I20250905 02:20:42.198861 15780 ts_itest-base.cc:119] --------------
2025-09-05T02:20:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T02:20:42Z Disabled control of system clock
I20250905 02:20:42.236289 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:36253
--webserver_interface=127.15.105.62
--webserver_port=0
--builtin_ntp_servers=127.15.105.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:36253
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250905 02:20:42.534157 16645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:42.534816 16645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:42.535296 16645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:42.566215 16645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:20:42.566526 16645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:42.566805 16645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:20:42.567054 16645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:20:42.602087 16645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:36253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:36253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:42.603345 16645 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:42.604925 16645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:42.614492 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:42.616266 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:43.832708 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:43.835253 16653 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1214 milliseconds
I20250905 02:20:43.835384 16645 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:43.836681 16645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:43.839996 16645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:43.841452 16645 hybrid_clock.cc:648] HybridClock initialized: now 1757038843841393 us; error 62 us; skew 500 ppm
I20250905 02:20:43.842442 16645 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:43.850091 16645 webserver.cc:480] Webserver started at http://127.15.105.62:42861/ using document root <none> and password file <none>
I20250905 02:20:43.851086 16645 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:43.851290 16645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:43.851773 16645 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:43.856328 16645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e2a13284d6164f3b8caaeb35ee954300"
format_stamp: "Formatted at 2025-09-05 02:20:43 on dist-test-slave-ctw4"
I20250905 02:20:43.857443 16645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e2a13284d6164f3b8caaeb35ee954300"
format_stamp: "Formatted at 2025-09-05 02:20:43 on dist-test-slave-ctw4"
I20250905 02:20:43.865190 16645 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250905 02:20:43.870957 16661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:43.872037 16645 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250905 02:20:43.872364 16645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e2a13284d6164f3b8caaeb35ee954300"
format_stamp: "Formatted at 2025-09-05 02:20:43 on dist-test-slave-ctw4"
I20250905 02:20:43.872699 16645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:43.924300 16645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:43.925729 16645 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:43.926167 16645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:43.994606 16645 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:36253
I20250905 02:20:43.994670 16712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:36253 every 8 connection(s)
I20250905 02:20:43.997259 16645 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:20:44.002753 16713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:44.005713 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 16645
I20250905 02:20:44.006090 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 02:20:44.026170 16713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300: Bootstrap starting.
I20250905 02:20:44.032594 16713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:44.034235 16713 log.cc:826] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:44.038558 16713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300: No bootstrap required, opened a new log
I20250905 02:20:44.056032 16713 raft_consensus.cc:357] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a13284d6164f3b8caaeb35ee954300" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 36253 } }
I20250905 02:20:44.056707 16713 raft_consensus.cc:383] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:44.056970 16713 raft_consensus.cc:738] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a13284d6164f3b8caaeb35ee954300, State: Initialized, Role: FOLLOWER
I20250905 02:20:44.057749 16713 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a13284d6164f3b8caaeb35ee954300" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 36253 } }
I20250905 02:20:44.058230 16713 raft_consensus.cc:397] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:44.058456 16713 raft_consensus.cc:491] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:44.058719 16713 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:44.062518 16713 raft_consensus.cc:513] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a13284d6164f3b8caaeb35ee954300" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 36253 } }
I20250905 02:20:44.063134 16713 leader_election.cc:304] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2a13284d6164f3b8caaeb35ee954300; no voters:
I20250905 02:20:44.064711 16713 leader_election.cc:290] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:44.065526 16718 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:44.068073 16718 raft_consensus.cc:695] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [term 1 LEADER]: Becoming Leader. State: Replica: e2a13284d6164f3b8caaeb35ee954300, State: Running, Role: LEADER
I20250905 02:20:44.068714 16718 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a13284d6164f3b8caaeb35ee954300" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 36253 } }
I20250905 02:20:44.069048 16713 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:44.078986 16720 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a13284d6164f3b8caaeb35ee954300. Latest consensus state: current_term: 1 leader_uuid: "e2a13284d6164f3b8caaeb35ee954300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a13284d6164f3b8caaeb35ee954300" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 36253 } } }
I20250905 02:20:44.079870 16720 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:44.081897 16719 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2a13284d6164f3b8caaeb35ee954300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a13284d6164f3b8caaeb35ee954300" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 36253 } } }
I20250905 02:20:44.082710 16719 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:44.085350 16726 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:20:44.097692 16726 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:44.113375 16726 catalog_manager.cc:1349] Generated new cluster ID: d5e62dcdc56241eeb8d100c3dc612b9f
I20250905 02:20:44.113705 16726 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:44.138988 16726 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:20:44.140586 16726 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:44.154302 16726 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e2a13284d6164f3b8caaeb35ee954300: Generated new TSK 0
I20250905 02:20:44.155215 16726 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:44.172942 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:0
--local_ip_for_outbound_sockets=127.15.105.1
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:36253
--builtin_ntp_servers=127.15.105.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250905 02:20:44.479110 16737 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250905 02:20:44.479735 16737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:44.480033 16737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:44.480502 16737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:44.512194 16737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:44.513057 16737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:20:44.549716 16737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:36253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:44.551016 16737 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:44.552611 16737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:44.565110 16743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:45.967846 16742 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 16737
W20250905 02:20:46.042330 16742 kernel_stack_watchdog.cc:198] Thread 16737 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:20:46.043587 16737 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.478s user 0.472s sys 0.912s
W20250905 02:20:46.043972 16745 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1476 milliseconds
W20250905 02:20:46.044068 16737 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.478s user 0.473s sys 0.912s
W20250905 02:20:44.567127 16744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:46.045583 16746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:46.045801 16737 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:46.049420 16737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:46.051957 16737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:46.053391 16737 hybrid_clock.cc:648] HybridClock initialized: now 1757038846053337 us; error 52 us; skew 500 ppm
I20250905 02:20:46.054428 16737 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:46.061800 16737 webserver.cc:480] Webserver started at http://127.15.105.1:42499/ using document root <none> and password file <none>
I20250905 02:20:46.063063 16737 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:46.063326 16737 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:46.063918 16737 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:46.070312 16737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "577296ff7d6547caa111993ae17f91d1"
format_stamp: "Formatted at 2025-09-05 02:20:46 on dist-test-slave-ctw4"
I20250905 02:20:46.071827 16737 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "577296ff7d6547caa111993ae17f91d1"
format_stamp: "Formatted at 2025-09-05 02:20:46 on dist-test-slave-ctw4"
I20250905 02:20:46.081143 16737 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20250905 02:20:46.088572 16753 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:46.089740 16737 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250905 02:20:46.090148 16737 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "577296ff7d6547caa111993ae17f91d1"
format_stamp: "Formatted at 2025-09-05 02:20:46 on dist-test-slave-ctw4"
I20250905 02:20:46.090567 16737 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:46.144130 16737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:46.145534 16737 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:46.145958 16737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:46.148550 16737 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:46.153184 16737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:46.153404 16737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:46.153657 16737 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:46.153812 16737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:46.321604 16737 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:45329
I20250905 02:20:46.321781 16865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:45329 every 8 connection(s)
I20250905 02:20:46.324199 16737 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:20:46.325134 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 16737
I20250905 02:20:46.325654 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 02:20:46.335430 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:0
--local_ip_for_outbound_sockets=127.15.105.2
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:36253
--builtin_ntp_servers=127.15.105.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250905 02:20:46.365584 16866 heartbeater.cc:344] Connected to a master server at 127.15.105.62:36253
I20250905 02:20:46.366022 16866 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:46.367044 16866 heartbeater.cc:507] Master 127.15.105.62:36253 requested a full tablet report, sending...
I20250905 02:20:46.369581 16678 ts_manager.cc:194] Registered new tserver with Master: 577296ff7d6547caa111993ae17f91d1 (127.15.105.1:45329)
I20250905 02:20:46.371639 16678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:42545
W20250905 02:20:46.698789 16870 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250905 02:20:46.699442 16870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:46.699710 16870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:46.700217 16870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:46.733037 16870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:46.733923 16870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:20:46.769744 16870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:36253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:46.771116 16870 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:46.772735 16870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:46.785706 16876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:47.375150 16866 heartbeater.cc:499] Master 127.15.105.62:36253 was elected leader, sending a full tablet report...
W20250905 02:20:46.790779 16879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:46.788065 16877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:48.010540 16878 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1220 milliseconds
I20250905 02:20:48.010640 16870 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:48.011871 16870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:48.014272 16870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:48.015769 16870 hybrid_clock.cc:648] HybridClock initialized: now 1757038848015626 us; error 87 us; skew 500 ppm
I20250905 02:20:48.016898 16870 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:48.023140 16870 webserver.cc:480] Webserver started at http://127.15.105.2:42335/ using document root <none> and password file <none>
I20250905 02:20:48.024169 16870 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:48.024380 16870 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:48.024825 16870 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:48.029439 16870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "668dee3d3e7047e2946b71313ca314ff"
format_stamp: "Formatted at 2025-09-05 02:20:48 on dist-test-slave-ctw4"
I20250905 02:20:48.030687 16870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "668dee3d3e7047e2946b71313ca314ff"
format_stamp: "Formatted at 2025-09-05 02:20:48 on dist-test-slave-ctw4"
I20250905 02:20:48.038936 16870 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250905 02:20:48.044452 16886 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:48.045430 16870 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250905 02:20:48.045743 16870 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "668dee3d3e7047e2946b71313ca314ff"
format_stamp: "Formatted at 2025-09-05 02:20:48 on dist-test-slave-ctw4"
I20250905 02:20:48.046088 16870 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:48.109333 16870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:48.110781 16870 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:48.111225 16870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:48.113698 16870 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:48.117705 16870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:48.117920 16870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:48.118185 16870 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:48.118343 16870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:48.252643 16870 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:35551
I20250905 02:20:48.252743 16998 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:35551 every 8 connection(s)
I20250905 02:20:48.255147 16870 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 02:20:48.261600 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 16870
I20250905 02:20:48.261996 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 02:20:48.267665 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:0
--local_ip_for_outbound_sockets=127.15.105.3
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:36253
--builtin_ntp_servers=127.15.105.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250905 02:20:48.276938 16999 heartbeater.cc:344] Connected to a master server at 127.15.105.62:36253
I20250905 02:20:48.277464 16999 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:48.278818 16999 heartbeater.cc:507] Master 127.15.105.62:36253 requested a full tablet report, sending...
I20250905 02:20:48.281262 16678 ts_manager.cc:194] Registered new tserver with Master: 668dee3d3e7047e2946b71313ca314ff (127.15.105.2:35551)
I20250905 02:20:48.282464 16678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:33437
W20250905 02:20:48.575438 17003 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250905 02:20:48.576083 17003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:48.576311 17003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:48.576742 17003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:48.609021 17003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:48.609992 17003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:20:48.645414 17003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:36253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:48.646641 17003 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:48.648200 17003 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:48.660792 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:49.286150 16999 heartbeater.cc:499] Master 127.15.105.62:36253 was elected leader, sending a full tablet report...
W20250905 02:20:48.661194 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:50.040822 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:50.043505 17003 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.383s user 0.506s sys 0.868s
W20250905 02:20:50.043926 17003 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.384s user 0.510s sys 0.868s
I20250905 02:20:50.045733 17003 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 02:20:50.045727 17011 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1379 milliseconds
I20250905 02:20:50.047403 17003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:50.049742 17003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:50.051126 17003 hybrid_clock.cc:648] HybridClock initialized: now 1757038850051095 us; error 46 us; skew 500 ppm
I20250905 02:20:50.051947 17003 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:50.059090 17003 webserver.cc:480] Webserver started at http://127.15.105.3:41193/ using document root <none> and password file <none>
I20250905 02:20:50.060143 17003 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:50.060365 17003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:50.060804 17003 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:50.065443 17003 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3e2d51a6b904490f9058643f7b413038"
format_stamp: "Formatted at 2025-09-05 02:20:50 on dist-test-slave-ctw4"
I20250905 02:20:50.066524 17003 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3e2d51a6b904490f9058643f7b413038"
format_stamp: "Formatted at 2025-09-05 02:20:50 on dist-test-slave-ctw4"
I20250905 02:20:50.074474 17003 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250905 02:20:50.081385 17019 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:50.082868 17003 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250905 02:20:50.083302 17003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3e2d51a6b904490f9058643f7b413038"
format_stamp: "Formatted at 2025-09-05 02:20:50 on dist-test-slave-ctw4"
I20250905 02:20:50.083698 17003 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:50.163055 17003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:50.164547 17003 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:50.164978 17003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:50.167488 17003 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:50.171603 17003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:50.171854 17003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250905 02:20:50.172094 17003 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:50.172251 17003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:50.311187 17003 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:46739
I20250905 02:20:50.311319 17131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:46739 every 8 connection(s)
I20250905 02:20:50.313695 17003 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 02:20:50.319938 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 17003
I20250905 02:20:50.320415 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 02:20:50.336386 17132 heartbeater.cc:344] Connected to a master server at 127.15.105.62:36253
I20250905 02:20:50.336818 17132 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:50.337919 17132 heartbeater.cc:507] Master 127.15.105.62:36253 requested a full tablet report, sending...
I20250905 02:20:50.340162 16678 ts_manager.cc:194] Registered new tserver with Master: 3e2d51a6b904490f9058643f7b413038 (127.15.105.3:46739)
I20250905 02:20:50.341524 16678 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:59571
I20250905 02:20:50.354892 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:20:50.385885 16678 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56686:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250905 02:20:50.405283 16678 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:20:50.466248 16934 tablet_service.cc:1468] Processing CreateTablet for tablet 0344189d1ca64621b72c4250ebf36b52 (DEFAULT_TABLE table=TestTable [id=d2ee5b26947149cc8b070dbe71a177bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:20:50.468760 16934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0344189d1ca64621b72c4250ebf36b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:50.472081 16801 tablet_service.cc:1468] Processing CreateTablet for tablet 0344189d1ca64621b72c4250ebf36b52 (DEFAULT_TABLE table=TestTable [id=d2ee5b26947149cc8b070dbe71a177bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:20:50.474004 16801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0344189d1ca64621b72c4250ebf36b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:50.475893 17067 tablet_service.cc:1468] Processing CreateTablet for tablet 0344189d1ca64621b72c4250ebf36b52 (DEFAULT_TABLE table=TestTable [id=d2ee5b26947149cc8b070dbe71a177bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:20:50.477257 17067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0344189d1ca64621b72c4250ebf36b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:50.498036 17152 tablet_bootstrap.cc:492] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038: Bootstrap starting.
I20250905 02:20:50.499603 17151 tablet_bootstrap.cc:492] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff: Bootstrap starting.
I20250905 02:20:50.502521 17153 tablet_bootstrap.cc:492] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1: Bootstrap starting.
I20250905 02:20:50.505987 17152 tablet_bootstrap.cc:654] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:50.507009 17151 tablet_bootstrap.cc:654] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:50.507951 17153 tablet_bootstrap.cc:654] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:50.508466 17152 log.cc:826] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:50.509464 17151 log.cc:826] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:50.510202 17153 log.cc:826] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:50.515246 17151 tablet_bootstrap.cc:492] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff: No bootstrap required, opened a new log
I20250905 02:20:50.515745 17151 ts_tablet_manager.cc:1397] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20250905 02:20:50.515974 17153 tablet_bootstrap.cc:492] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1: No bootstrap required, opened a new log
I20250905 02:20:50.516088 17152 tablet_bootstrap.cc:492] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038: No bootstrap required, opened a new log
I20250905 02:20:50.516422 17153 ts_tablet_manager.cc:1397] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20250905 02:20:50.516579 17152 ts_tablet_manager.cc:1397] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20250905 02:20:50.535161 17153 raft_consensus.cc:357] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:50.536072 17153 raft_consensus.cc:738] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577296ff7d6547caa111993ae17f91d1, State: Initialized, Role: FOLLOWER
I20250905 02:20:50.536667 17153 consensus_queue.cc:260] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:50.540580 17153 ts_tablet_manager.cc:1428] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1: Time spent starting tablet: real 0.024s user 0.021s sys 0.003s
I20250905 02:20:50.542932 17151 raft_consensus.cc:357] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:50.543572 17151 raft_consensus.cc:738] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668dee3d3e7047e2946b71313ca314ff, State: Initialized, Role: FOLLOWER
I20250905 02:20:50.544164 17151 consensus_queue.cc:260] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:50.544574 17152 raft_consensus.cc:357] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:50.545480 17152 raft_consensus.cc:738] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2d51a6b904490f9058643f7b413038, State: Initialized, Role: FOLLOWER
I20250905 02:20:50.546187 17152 consensus_queue.cc:260] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:50.549741 17132 heartbeater.cc:499] Master 127.15.105.62:36253 was elected leader, sending a full tablet report...
I20250905 02:20:50.552467 17151 ts_tablet_manager.cc:1428] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff: Time spent starting tablet: real 0.036s user 0.027s sys 0.005s
I20250905 02:20:50.554509 17152 ts_tablet_manager.cc:1428] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038: Time spent starting tablet: real 0.038s user 0.019s sys 0.013s
W20250905 02:20:50.568181 17133 tablet.cc:2378] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 02:20:50.584777 16867 tablet.cc:2378] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:20:50.586226 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:20:50.589598 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 577296ff7d6547caa111993ae17f91d1 to finish bootstrapping
I20250905 02:20:50.602619 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 668dee3d3e7047e2946b71313ca314ff to finish bootstrapping
I20250905 02:20:50.613529 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3e2d51a6b904490f9058643f7b413038 to finish bootstrapping
I20250905 02:20:50.655280 16821 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0344189d1ca64621b72c4250ebf36b52"
dest_uuid: "577296ff7d6547caa111993ae17f91d1"
from {username='slave'} at 127.0.0.1:56290
I20250905 02:20:50.655869 16821 raft_consensus.cc:491] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 02:20:50.656173 16821 raft_consensus.cc:3058] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:50.660673 16821 raft_consensus.cc:513] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:50.662921 16821 leader_election.cc:290] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [CANDIDATE]: Term 1 election: Requested vote from peers 668dee3d3e7047e2946b71313ca314ff (127.15.105.2:35551), 3e2d51a6b904490f9058643f7b413038 (127.15.105.3:46739)
I20250905 02:20:50.674604 15780 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250905 02:20:50.676400 16954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0344189d1ca64621b72c4250ebf36b52" candidate_uuid: "577296ff7d6547caa111993ae17f91d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "668dee3d3e7047e2946b71313ca314ff"
I20250905 02:20:50.677007 16954 raft_consensus.cc:3058] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:50.678139 17087 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0344189d1ca64621b72c4250ebf36b52" candidate_uuid: "577296ff7d6547caa111993ae17f91d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3e2d51a6b904490f9058643f7b413038"
I20250905 02:20:50.678699 17087 raft_consensus.cc:3058] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:50.681649 16954 raft_consensus.cc:2466] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 577296ff7d6547caa111993ae17f91d1 in term 1.
I20250905 02:20:50.682750 16754 leader_election.cc:304] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 577296ff7d6547caa111993ae17f91d1, 668dee3d3e7047e2946b71313ca314ff; no voters:
I20250905 02:20:50.683024 17087 raft_consensus.cc:2466] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 577296ff7d6547caa111993ae17f91d1 in term 1.
I20250905 02:20:50.683557 17157 raft_consensus.cc:2802] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:50.685245 17157 raft_consensus.cc:695] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 1 LEADER]: Becoming Leader. State: Replica: 577296ff7d6547caa111993ae17f91d1, State: Running, Role: LEADER
I20250905 02:20:50.685992 17157 consensus_queue.cc:237] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:50.697571 16678 catalog_manager.cc:5582] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 577296ff7d6547caa111993ae17f91d1 (127.15.105.1). New cstate: current_term: 1 leader_uuid: "577296ff7d6547caa111993ae17f91d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } health_report { overall_health: UNKNOWN } } }
W20250905 02:20:50.761875 17000 tablet.cc:2378] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:20:50.779873 15780 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250905 02:20:50.984992 15780 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250905 02:20:51.080510 17157 consensus_queue.cc:1035] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:51.096081 17157 consensus_queue.cc:1035] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:52.992153 16821 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0344189d1ca64621b72c4250ebf36b52"
dest_uuid: "577296ff7d6547caa111993ae17f91d1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56302
I20250905 02:20:52.992623 16821 raft_consensus.cc:604] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 1 LEADER]: Received request to transfer leadership
I20250905 02:20:53.129789 17195 raft_consensus.cc:991] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1: : Instructing follower 3e2d51a6b904490f9058643f7b413038 to start an election
I20250905 02:20:53.130318 17178 raft_consensus.cc:1079] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 1 LEADER]: Signalling peer 3e2d51a6b904490f9058643f7b413038 to start an election
I20250905 02:20:53.131702 17087 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0344189d1ca64621b72c4250ebf36b52"
dest_uuid: "3e2d51a6b904490f9058643f7b413038"
from {username='slave'} at 127.15.105.1:56355
I20250905 02:20:53.132243 17087 raft_consensus.cc:491] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 02:20:53.132490 17087 raft_consensus.cc:3058] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:53.136524 17087 raft_consensus.cc:513] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:53.138521 17087 leader_election.cc:290] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [CANDIDATE]: Term 2 election: Requested vote from peers 668dee3d3e7047e2946b71313ca314ff (127.15.105.2:35551), 577296ff7d6547caa111993ae17f91d1 (127.15.105.1:45329)
I20250905 02:20:53.149809 16821 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0344189d1ca64621b72c4250ebf36b52" candidate_uuid: "3e2d51a6b904490f9058643f7b413038" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "577296ff7d6547caa111993ae17f91d1"
I20250905 02:20:53.150359 16821 raft_consensus.cc:3053] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 02:20:53.150592 16821 raft_consensus.cc:738] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 577296ff7d6547caa111993ae17f91d1, State: Running, Role: LEADER
I20250905 02:20:53.150781 16954 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0344189d1ca64621b72c4250ebf36b52" candidate_uuid: "3e2d51a6b904490f9058643f7b413038" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "668dee3d3e7047e2946b71313ca314ff"
I20250905 02:20:53.151234 16954 raft_consensus.cc:3058] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:53.151060 16821 consensus_queue.cc:260] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:53.151998 16821 raft_consensus.cc:3058] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:53.155189 16954 raft_consensus.cc:2466] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2d51a6b904490f9058643f7b413038 in term 2.
I20250905 02:20:53.155939 16821 raft_consensus.cc:2466] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2d51a6b904490f9058643f7b413038 in term 2.
I20250905 02:20:53.156234 17020 leader_election.cc:304] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2d51a6b904490f9058643f7b413038, 668dee3d3e7047e2946b71313ca314ff; no voters:
I20250905 02:20:53.158059 17199 raft_consensus.cc:2802] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:20:53.159317 17199 raft_consensus.cc:695] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [term 2 LEADER]: Becoming Leader. State: Replica: 3e2d51a6b904490f9058643f7b413038, State: Running, Role: LEADER
I20250905 02:20:53.160051 17199 consensus_queue.cc:237] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } }
I20250905 02:20:53.168243 16676 catalog_manager.cc:5582] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 reported cstate change: term changed from 1 to 2, leader changed from 577296ff7d6547caa111993ae17f91d1 (127.15.105.1) to 3e2d51a6b904490f9058643f7b413038 (127.15.105.3). New cstate: current_term: 2 leader_uuid: "3e2d51a6b904490f9058643f7b413038" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e2d51a6b904490f9058643f7b413038" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 46739 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:53.547749 16954 raft_consensus.cc:1273] T 0344189d1ca64621b72c4250ebf36b52 P 668dee3d3e7047e2946b71313ca314ff [term 2 FOLLOWER]: Refusing update from remote peer 3e2d51a6b904490f9058643f7b413038: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:20:53.549024 17199 consensus_queue.cc:1035] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "668dee3d3e7047e2946b71313ca314ff" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:20:53.560781 16821 raft_consensus.cc:1273] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 2 FOLLOWER]: Refusing update from remote peer 3e2d51a6b904490f9058643f7b413038: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:20:53.562306 17199 consensus_queue.cc:1035] T 0344189d1ca64621b72c4250ebf36b52 P 3e2d51a6b904490f9058643f7b413038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "577296ff7d6547caa111993ae17f91d1" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:20:55.717023 16821 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0344189d1ca64621b72c4250ebf36b52"
dest_uuid: "577296ff7d6547caa111993ae17f91d1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56306
I20250905 02:20:55.717594 16821 raft_consensus.cc:604] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 2 FOLLOWER]: Received request to transfer leadership
I20250905 02:20:55.717911 16821 raft_consensus.cc:612] T 0344189d1ca64621b72c4250ebf36b52 P 577296ff7d6547caa111993ae17f91d1 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 02:20:56.758941 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 16737
I20250905 02:20:56.783977 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 16870
I20250905 02:20:56.806913 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 17003
I20250905 02:20:56.836437 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 16645
2025-09-05T02:20:56Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14700 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250905 02:20:56.892930 15780 test_util.cc:276] Using random seed: 2095837670
I20250905 02:20:56.896957 15780 ts_itest-base.cc:115] Starting cluster with:
I20250905 02:20:56.897128 15780 ts_itest-base.cc:116] --------------
I20250905 02:20:56.897306 15780 ts_itest-base.cc:117] 3 tablet servers
I20250905 02:20:56.897450 15780 ts_itest-base.cc:118] 3 replicas per TS
I20250905 02:20:56.897589 15780 ts_itest-base.cc:119] --------------
2025-09-05T02:20:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T02:20:56Z Disabled control of system clock
I20250905 02:20:56.934412 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:41853
--webserver_interface=127.15.105.62
--webserver_port=0
--builtin_ntp_servers=127.15.105.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:41853 with env {}
W20250905 02:20:57.233026 17243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:57.233575 17243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:57.233989 17243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:57.264678 17243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:20:57.264987 17243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:57.265278 17243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:20:57.265520 17243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:20:57.300558 17243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:41853
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:41853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:57.301838 17243 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:57.303401 17243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:57.313642 17249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:58.717794 17248 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 17243
W20250905 02:20:57.314080 17250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:59.117403 17243 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.803s user 0.778s sys 1.023s
W20250905 02:20:59.117889 17243 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.804s user 0.779s sys 1.024s
W20250905 02:20:59.119671 17252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:59.122112 17251 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1804 milliseconds
I20250905 02:20:59.122205 17243 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:20:59.123545 17243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:20:59.125948 17243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:20:59.127344 17243 hybrid_clock.cc:648] HybridClock initialized: now 1757038859127297 us; error 64 us; skew 500 ppm
I20250905 02:20:59.128182 17243 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:59.134542 17243 webserver.cc:480] Webserver started at http://127.15.105.62:45671/ using document root <none> and password file <none>
I20250905 02:20:59.135494 17243 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:59.135721 17243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:59.136225 17243 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:59.141970 17243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "579c098b93ed4a7eba14a371ffda9955"
format_stamp: "Formatted at 2025-09-05 02:20:59 on dist-test-slave-ctw4"
I20250905 02:20:59.143113 17243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "579c098b93ed4a7eba14a371ffda9955"
format_stamp: "Formatted at 2025-09-05 02:20:59 on dist-test-slave-ctw4"
I20250905 02:20:59.150633 17243 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.003s
I20250905 02:20:59.156370 17259 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:59.157454 17243 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250905 02:20:59.157800 17243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "579c098b93ed4a7eba14a371ffda9955"
format_stamp: "Formatted at 2025-09-05 02:20:59 on dist-test-slave-ctw4"
I20250905 02:20:59.158155 17243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:59.210851 17243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:59.212917 17243 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:59.213389 17243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:59.285006 17243 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:41853
I20250905 02:20:59.285171 17310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:41853 every 8 connection(s)
I20250905 02:20:59.287818 17243 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:20:59.293879 17311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:59.295327 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 17243
I20250905 02:20:59.295816 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 02:20:59.316920 17311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955: Bootstrap starting.
I20250905 02:20:59.322973 17311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:59.325242 17311 log.cc:826] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:59.329591 17311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955: No bootstrap required, opened a new log
I20250905 02:20:59.346412 17311 raft_consensus.cc:357] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579c098b93ed4a7eba14a371ffda9955" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41853 } }
I20250905 02:20:59.347045 17311 raft_consensus.cc:383] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:59.347333 17311 raft_consensus.cc:738] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 579c098b93ed4a7eba14a371ffda9955, State: Initialized, Role: FOLLOWER
I20250905 02:20:59.348058 17311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579c098b93ed4a7eba14a371ffda9955" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41853 } }
I20250905 02:20:59.348563 17311 raft_consensus.cc:397] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:59.348831 17311 raft_consensus.cc:491] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:59.349130 17311 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:59.353569 17311 raft_consensus.cc:513] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579c098b93ed4a7eba14a371ffda9955" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41853 } }
I20250905 02:20:59.354214 17311 leader_election.cc:304] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 579c098b93ed4a7eba14a371ffda9955; no voters:
I20250905 02:20:59.355829 17311 leader_election.cc:290] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:59.356570 17316 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:59.358716 17316 raft_consensus.cc:695] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [term 1 LEADER]: Becoming Leader. State: Replica: 579c098b93ed4a7eba14a371ffda9955, State: Running, Role: LEADER
I20250905 02:20:59.359565 17316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579c098b93ed4a7eba14a371ffda9955" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41853 } }
I20250905 02:20:59.360766 17311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:59.371044 17318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 579c098b93ed4a7eba14a371ffda9955. Latest consensus state: current_term: 1 leader_uuid: "579c098b93ed4a7eba14a371ffda9955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579c098b93ed4a7eba14a371ffda9955" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41853 } } }
I20250905 02:20:59.371984 17318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:59.373207 17317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "579c098b93ed4a7eba14a371ffda9955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "579c098b93ed4a7eba14a371ffda9955" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41853 } } }
I20250905 02:20:59.374028 17317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:59.375548 17325 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:20:59.385802 17325 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:59.404821 17325 catalog_manager.cc:1349] Generated new cluster ID: ebb61e84d3464378bf4a5098fe97bb18
I20250905 02:20:59.405089 17325 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:59.419387 17325 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:20:59.420887 17325 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:59.435050 17325 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 579c098b93ed4a7eba14a371ffda9955: Generated new TSK 0
I20250905 02:20:59.436259 17325 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:59.454198 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:0
--local_ip_for_outbound_sockets=127.15.105.1
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41853
--builtin_ntp_servers=127.15.105.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 02:20:59.761066 17335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:20:59.761574 17335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:20:59.762076 17335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:20:59.794672 17335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:20:59.795722 17335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:20:59.831773 17335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41853
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:20:59.833101 17335 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:59.834659 17335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:59.847213 17341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:59.848932 17342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:01.250553 17340 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 17335
W20250905 02:21:01.362696 17335 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.514s user 0.462s sys 1.050s
W20250905 02:21:01.363066 17335 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.515s user 0.462s sys 1.050s
W20250905 02:21:01.365022 17344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:01.369841 17343 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1515 milliseconds
I20250905 02:21:01.369935 17335 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:01.371203 17335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:01.373793 17335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:01.375299 17335 hybrid_clock.cc:648] HybridClock initialized: now 1757038861375253 us; error 67 us; skew 500 ppm
I20250905 02:21:01.376159 17335 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:01.383044 17335 webserver.cc:480] Webserver started at http://127.15.105.1:39453/ using document root <none> and password file <none>
I20250905 02:21:01.384079 17335 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:01.384315 17335 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:01.384749 17335 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:01.389302 17335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2a9696d317814d818a702bcf4e131926"
format_stamp: "Formatted at 2025-09-05 02:21:01 on dist-test-slave-ctw4"
I20250905 02:21:01.390489 17335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2a9696d317814d818a702bcf4e131926"
format_stamp: "Formatted at 2025-09-05 02:21:01 on dist-test-slave-ctw4"
I20250905 02:21:01.398113 17335 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.009s
I20250905 02:21:01.404402 17351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:01.405486 17335 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250905 02:21:01.405778 17335 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2a9696d317814d818a702bcf4e131926"
format_stamp: "Formatted at 2025-09-05 02:21:01 on dist-test-slave-ctw4"
I20250905 02:21:01.406073 17335 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:01.470659 17335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:01.472190 17335 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:01.472733 17335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:01.475687 17335 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:01.480561 17335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:01.480767 17335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:01.480973 17335 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:01.481107 17335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:01.658627 17335 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:39405
I20250905 02:21:01.658810 17463 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:39405 every 8 connection(s)
I20250905 02:21:01.661273 17335 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:21:01.667021 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 17335
I20250905 02:21:01.667537 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 02:21:01.677448 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:0
--local_ip_for_outbound_sockets=127.15.105.2
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41853
--builtin_ntp_servers=127.15.105.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:01.701097 17464 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41853
I20250905 02:21:01.701649 17464 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:01.703024 17464 heartbeater.cc:507] Master 127.15.105.62:41853 requested a full tablet report, sending...
I20250905 02:21:01.706475 17276 ts_manager.cc:194] Registered new tserver with Master: 2a9696d317814d818a702bcf4e131926 (127.15.105.1:39405)
I20250905 02:21:01.709219 17276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:52487
W20250905 02:21:01.993660 17468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:01.994201 17468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:01.994832 17468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:02.034000 17468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:02.034884 17468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:21:02.070619 17468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41853
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:02.071992 17468 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:02.073575 17468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:02.085335 17474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:02.713646 17464 heartbeater.cc:499] Master 127.15.105.62:41853 was elected leader, sending a full tablet report...
W20250905 02:21:02.085996 17475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:03.361505 17477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:03.363826 17476 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1273 milliseconds
I20250905 02:21:03.363931 17468 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:03.365109 17468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:03.367204 17468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:03.368526 17468 hybrid_clock.cc:648] HybridClock initialized: now 1757038863368505 us; error 39 us; skew 500 ppm
I20250905 02:21:03.369282 17468 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:03.375706 17468 webserver.cc:480] Webserver started at http://127.15.105.2:41037/ using document root <none> and password file <none>
I20250905 02:21:03.376642 17468 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:03.376827 17468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:03.377238 17468 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:03.381691 17468 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "430e897ab5954d82b5bd92ef2b1972fd"
format_stamp: "Formatted at 2025-09-05 02:21:03 on dist-test-slave-ctw4"
I20250905 02:21:03.382756 17468 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "430e897ab5954d82b5bd92ef2b1972fd"
format_stamp: "Formatted at 2025-09-05 02:21:03 on dist-test-slave-ctw4"
I20250905 02:21:03.389686 17468 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250905 02:21:03.395678 17485 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:03.397106 17468 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250905 02:21:03.397511 17468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "430e897ab5954d82b5bd92ef2b1972fd"
format_stamp: "Formatted at 2025-09-05 02:21:03 on dist-test-slave-ctw4"
I20250905 02:21:03.397930 17468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:03.469074 17468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:03.470500 17468 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:03.470897 17468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:03.473307 17468 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:03.477159 17468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:03.477356 17468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:03.477545 17468 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:03.477679 17468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:03.610942 17468 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:35485
I20250905 02:21:03.611042 17597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:35485 every 8 connection(s)
I20250905 02:21:03.613540 17468 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 02:21:03.620471 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 17468
I20250905 02:21:03.621168 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 02:21:03.628484 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:0
--local_ip_for_outbound_sockets=127.15.105.3
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41853
--builtin_ntp_servers=127.15.105.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:03.636574 17598 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41853
I20250905 02:21:03.637058 17598 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:03.638154 17598 heartbeater.cc:507] Master 127.15.105.62:41853 requested a full tablet report, sending...
I20250905 02:21:03.640595 17276 ts_manager.cc:194] Registered new tserver with Master: 430e897ab5954d82b5bd92ef2b1972fd (127.15.105.2:35485)
I20250905 02:21:03.641940 17276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:36157
W20250905 02:21:03.752419 17307 debug-util.cc:398] Leaking SignalData structure 0x7b080009f4a0 after lost signal to thread 17244
W20250905 02:21:03.939495 17602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:03.940074 17602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:03.940583 17602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:03.973543 17602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:03.974390 17602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:21:04.010702 17602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41853
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:04.012032 17602 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:04.014122 17602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:04.028209 17608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:04.645428 17598 heartbeater.cc:499] Master 127.15.105.62:41853 was elected leader, sending a full tablet report...
W20250905 02:21:04.028781 17609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:05.361972 17602 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.335s user 0.482s sys 0.846s
W20250905 02:21:05.362399 17602 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.336s user 0.483s sys 0.847s
I20250905 02:21:05.369288 17602 server_base.cc:1047] running on GCE node
W20250905 02:21:05.370703 17613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:05.372337 17602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:05.375278 17602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:05.376901 17602 hybrid_clock.cc:648] HybridClock initialized: now 1757038865376840 us; error 59 us; skew 500 ppm
I20250905 02:21:05.378064 17602 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:05.388799 17602 webserver.cc:480] Webserver started at http://127.15.105.3:34659/ using document root <none> and password file <none>
I20250905 02:21:05.390347 17602 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:05.390673 17602 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:05.391289 17602 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:05.398918 17602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "901667520b6d440bb63687299588df60"
format_stamp: "Formatted at 2025-09-05 02:21:05 on dist-test-slave-ctw4"
I20250905 02:21:05.400756 17602 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "901667520b6d440bb63687299588df60"
format_stamp: "Formatted at 2025-09-05 02:21:05 on dist-test-slave-ctw4"
I20250905 02:21:05.411444 17602 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.005s sys 0.005s
I20250905 02:21:05.419705 17618 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:05.421213 17602 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250905 02:21:05.421635 17602 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "901667520b6d440bb63687299588df60"
format_stamp: "Formatted at 2025-09-05 02:21:05 on dist-test-slave-ctw4"
I20250905 02:21:05.422087 17602 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:05.513530 17602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:05.514899 17602 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:05.515307 17602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:05.518321 17602 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:05.522783 17602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:05.523072 17602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:05.523377 17602 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:05.523634 17602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250905 02:21:05.690480 17602 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:38045
I20250905 02:21:05.690518 17730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:38045 every 8 connection(s)
I20250905 02:21:05.694375 17602 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 02:21:05.697049 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 17602
I20250905 02:21:05.697547 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 02:21:05.729449 17731 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41853
I20250905 02:21:05.729969 17731 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:05.731309 17731 heartbeater.cc:507] Master 127.15.105.62:41853 requested a full tablet report, sending...
I20250905 02:21:05.733675 17276 ts_manager.cc:194] Registered new tserver with Master: 901667520b6d440bb63687299588df60 (127.15.105.3:38045)
I20250905 02:21:05.735471 17276 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:39343
I20250905 02:21:05.747337 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:21:05.784065 17276 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36502:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250905 02:21:05.803651 17276 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:21:05.885505 17533 tablet_service.cc:1468] Processing CreateTablet for tablet 92778bb37ab94ca096dfecdcc7c5e8b2 (DEFAULT_TABLE table=TestTable [id=36c56b6b17534cf1b05ddce3d3c6e1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:05.887454 17533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92778bb37ab94ca096dfecdcc7c5e8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:05.891727 17399 tablet_service.cc:1468] Processing CreateTablet for tablet 92778bb37ab94ca096dfecdcc7c5e8b2 (DEFAULT_TABLE table=TestTable [id=36c56b6b17534cf1b05ddce3d3c6e1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:05.893870 17399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92778bb37ab94ca096dfecdcc7c5e8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:05.893373 17666 tablet_service.cc:1468] Processing CreateTablet for tablet 92778bb37ab94ca096dfecdcc7c5e8b2 (DEFAULT_TABLE table=TestTable [id=36c56b6b17534cf1b05ddce3d3c6e1c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:05.895339 17666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92778bb37ab94ca096dfecdcc7c5e8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:05.919796 17750 tablet_bootstrap.cc:492] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd: Bootstrap starting.
I20250905 02:21:05.929005 17750 tablet_bootstrap.cc:654] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:05.930975 17751 tablet_bootstrap.cc:492] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926: Bootstrap starting.
I20250905 02:21:05.931519 17750 log.cc:826] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:05.934743 17752 tablet_bootstrap.cc:492] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60: Bootstrap starting.
I20250905 02:21:05.938460 17750 tablet_bootstrap.cc:492] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd: No bootstrap required, opened a new log
I20250905 02:21:05.938977 17750 ts_tablet_manager.cc:1397] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20250905 02:21:05.940788 17751 tablet_bootstrap.cc:654] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:05.942361 17752 tablet_bootstrap.cc:654] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:05.943302 17751 log.cc:826] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:05.944669 17752 log.cc:826] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:05.953033 17752 tablet_bootstrap.cc:492] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60: No bootstrap required, opened a new log
I20250905 02:21:05.953469 17752 ts_tablet_manager.cc:1397] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.008s
I20250905 02:21:05.956103 17751 tablet_bootstrap.cc:492] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926: No bootstrap required, opened a new log
I20250905 02:21:05.956621 17751 ts_tablet_manager.cc:1397] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.004s
I20250905 02:21:05.970726 17750 raft_consensus.cc:357] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:05.971671 17750 raft_consensus.cc:383] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:05.972028 17750 raft_consensus.cc:738] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430e897ab5954d82b5bd92ef2b1972fd, State: Initialized, Role: FOLLOWER
I20250905 02:21:05.973022 17750 consensus_queue.cc:260] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:05.980360 17752 raft_consensus.cc:357] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:05.981400 17752 raft_consensus.cc:383] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:05.981746 17752 raft_consensus.cc:738] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901667520b6d440bb63687299588df60, State: Initialized, Role: FOLLOWER
I20250905 02:21:05.982784 17752 consensus_queue.cc:260] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:05.983712 17751 raft_consensus.cc:357] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:05.984701 17751 raft_consensus.cc:383] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:05.985186 17751 raft_consensus.cc:738] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9696d317814d818a702bcf4e131926, State: Initialized, Role: FOLLOWER
I20250905 02:21:05.985968 17750 ts_tablet_manager.cc:1428] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd: Time spent starting tablet: real 0.047s user 0.036s sys 0.003s
I20250905 02:21:05.986285 17751 consensus_queue.cc:260] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:05.988898 17731 heartbeater.cc:499] Master 127.15.105.62:41853 was elected leader, sending a full tablet report...
I20250905 02:21:05.990398 17752 ts_tablet_manager.cc:1428] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60: Time spent starting tablet: real 0.037s user 0.024s sys 0.007s
I20250905 02:21:05.996186 17751 ts_tablet_manager.cc:1428] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926: Time spent starting tablet: real 0.039s user 0.029s sys 0.003s
W20250905 02:21:06.120584 17599 tablet.cc:2378] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 02:21:06.171335 17465 tablet.cc:2378] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 02:21:06.206708 17732 tablet.cc:2378] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:21:06.209785 17756 raft_consensus.cc:491] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:21:06.210214 17756 raft_consensus.cc:513] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:06.212570 17756 leader_election.cc:290] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 901667520b6d440bb63687299588df60 (127.15.105.3:38045), 2a9696d317814d818a702bcf4e131926 (127.15.105.1:39405)
I20250905 02:21:06.225601 17419 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92778bb37ab94ca096dfecdcc7c5e8b2" candidate_uuid: "430e897ab5954d82b5bd92ef2b1972fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9696d317814d818a702bcf4e131926" is_pre_election: true
I20250905 02:21:06.225771 17686 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92778bb37ab94ca096dfecdcc7c5e8b2" candidate_uuid: "430e897ab5954d82b5bd92ef2b1972fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901667520b6d440bb63687299588df60" is_pre_election: true
I20250905 02:21:06.226318 17419 raft_consensus.cc:2466] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430e897ab5954d82b5bd92ef2b1972fd in term 0.
I20250905 02:21:06.226445 17686 raft_consensus.cc:2466] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430e897ab5954d82b5bd92ef2b1972fd in term 0.
I20250905 02:21:06.227454 17486 leader_election.cc:304] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9696d317814d818a702bcf4e131926, 430e897ab5954d82b5bd92ef2b1972fd; no voters:
I20250905 02:21:06.228154 17756 raft_consensus.cc:2802] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:21:06.228505 17756 raft_consensus.cc:491] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:21:06.228746 17756 raft_consensus.cc:3058] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:06.233014 17756 raft_consensus.cc:513] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:06.234300 17756 leader_election.cc:290] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [CANDIDATE]: Term 1 election: Requested vote from peers 901667520b6d440bb63687299588df60 (127.15.105.3:38045), 2a9696d317814d818a702bcf4e131926 (127.15.105.1:39405)
I20250905 02:21:06.234947 17686 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92778bb37ab94ca096dfecdcc7c5e8b2" candidate_uuid: "430e897ab5954d82b5bd92ef2b1972fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901667520b6d440bb63687299588df60"
I20250905 02:21:06.235189 17419 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92778bb37ab94ca096dfecdcc7c5e8b2" candidate_uuid: "430e897ab5954d82b5bd92ef2b1972fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9696d317814d818a702bcf4e131926"
I20250905 02:21:06.235453 17686 raft_consensus.cc:3058] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:06.235687 17419 raft_consensus.cc:3058] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:06.242303 17419 raft_consensus.cc:2466] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 2a9696d317814d818a702bcf4e131926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430e897ab5954d82b5bd92ef2b1972fd in term 1.
I20250905 02:21:06.242303 17686 raft_consensus.cc:2466] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 901667520b6d440bb63687299588df60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430e897ab5954d82b5bd92ef2b1972fd in term 1.
I20250905 02:21:06.243193 17486 leader_election.cc:304] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9696d317814d818a702bcf4e131926, 430e897ab5954d82b5bd92ef2b1972fd; no voters:
I20250905 02:21:06.243737 17756 raft_consensus.cc:2802] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:06.245244 17756 raft_consensus.cc:695] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [term 1 LEADER]: Becoming Leader. State: Replica: 430e897ab5954d82b5bd92ef2b1972fd, State: Running, Role: LEADER
I20250905 02:21:06.246057 17756 consensus_queue.cc:237] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } }
I20250905 02:21:06.256378 17274 catalog_manager.cc:5582] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 430e897ab5954d82b5bd92ef2b1972fd (127.15.105.2). New cstate: current_term: 1 leader_uuid: "430e897ab5954d82b5bd92ef2b1972fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } health_report { overall_health: UNKNOWN } } }
I20250905 02:21:06.313355 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:21:06.316525 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2a9696d317814d818a702bcf4e131926 to finish bootstrapping
I20250905 02:21:06.327488 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 430e897ab5954d82b5bd92ef2b1972fd to finish bootstrapping
I20250905 02:21:06.337756 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 901667520b6d440bb63687299588df60 to finish bootstrapping
I20250905 02:21:06.349977 17274 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36502:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250905 02:21:06.351392 17274 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:21:06.368144 17399 tablet_service.cc:1468] Processing CreateTablet for tablet 6db6c9804fe84295835bc5fe09776ec0 (DEFAULT_TABLE table=TestAnotherTable [id=2257226690d14470a8b03981925ece25]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250905 02:21:06.368402 17533 tablet_service.cc:1468] Processing CreateTablet for tablet 6db6c9804fe84295835bc5fe09776ec0 (DEFAULT_TABLE table=TestAnotherTable [id=2257226690d14470a8b03981925ece25]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250905 02:21:06.369237 17399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db6c9804fe84295835bc5fe09776ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:06.369455 17533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db6c9804fe84295835bc5fe09776ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:06.369228 17666 tablet_service.cc:1468] Processing CreateTablet for tablet 6db6c9804fe84295835bc5fe09776ec0 (DEFAULT_TABLE table=TestAnotherTable [id=2257226690d14470a8b03981925ece25]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250905 02:21:06.370281 17666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6db6c9804fe84295835bc5fe09776ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:06.381719 17751 tablet_bootstrap.cc:492] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926: Bootstrap starting.
I20250905 02:21:06.383447 17750 tablet_bootstrap.cc:492] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd: Bootstrap starting.
I20250905 02:21:06.384478 17752 tablet_bootstrap.cc:492] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60: Bootstrap starting.
I20250905 02:21:06.388190 17751 tablet_bootstrap.cc:654] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:06.389451 17750 tablet_bootstrap.cc:654] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:06.390609 17752 tablet_bootstrap.cc:654] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:06.394585 17751 tablet_bootstrap.cc:492] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926: No bootstrap required, opened a new log
I20250905 02:21:06.394915 17751 ts_tablet_manager.cc:1397] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20250905 02:21:06.397326 17751 raft_consensus.cc:357] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.398087 17751 raft_consensus.cc:383] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:06.398420 17751 raft_consensus.cc:738] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9696d317814d818a702bcf4e131926, State: Initialized, Role: FOLLOWER
I20250905 02:21:06.399085 17751 consensus_queue.cc:260] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.401698 17752 tablet_bootstrap.cc:492] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60: No bootstrap required, opened a new log
I20250905 02:21:06.401723 17750 tablet_bootstrap.cc:492] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd: No bootstrap required, opened a new log
I20250905 02:21:06.402187 17752 ts_tablet_manager.cc:1397] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20250905 02:21:06.402189 17750 ts_tablet_manager.cc:1397] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20250905 02:21:06.402276 17751 ts_tablet_manager.cc:1428] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 02:21:06.404340 17750 raft_consensus.cc:357] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.404879 17752 raft_consensus.cc:357] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.405470 17750 raft_consensus.cc:383] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:06.405591 17752 raft_consensus.cc:383] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:06.405820 17750 raft_consensus.cc:738] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 430e897ab5954d82b5bd92ef2b1972fd, State: Initialized, Role: FOLLOWER
I20250905 02:21:06.405910 17752 raft_consensus.cc:738] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901667520b6d440bb63687299588df60, State: Initialized, Role: FOLLOWER
I20250905 02:21:06.406484 17750 consensus_queue.cc:260] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.407464 17752 consensus_queue.cc:260] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.411201 17752 ts_tablet_manager.cc:1428] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20250905 02:21:06.412990 17750 ts_tablet_manager.cc:1428] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250905 02:21:06.477173 17756 raft_consensus.cc:491] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:21:06.477579 17756 raft_consensus.cc:513] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.479154 17756 leader_election.cc:290] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a9696d317814d818a702bcf4e131926 (127.15.105.1:39405), 901667520b6d440bb63687299588df60 (127.15.105.3:38045)
I20250905 02:21:06.479993 17419 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db6c9804fe84295835bc5fe09776ec0" candidate_uuid: "430e897ab5954d82b5bd92ef2b1972fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9696d317814d818a702bcf4e131926" is_pre_election: true
I20250905 02:21:06.480139 17686 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db6c9804fe84295835bc5fe09776ec0" candidate_uuid: "430e897ab5954d82b5bd92ef2b1972fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901667520b6d440bb63687299588df60" is_pre_election: true
I20250905 02:21:06.480567 17419 raft_consensus.cc:2466] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430e897ab5954d82b5bd92ef2b1972fd in term 0.
I20250905 02:21:06.480751 17686 raft_consensus.cc:2466] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 430e897ab5954d82b5bd92ef2b1972fd in term 0.
I20250905 02:21:06.481478 17486 leader_election.cc:304] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9696d317814d818a702bcf4e131926, 430e897ab5954d82b5bd92ef2b1972fd; no voters:
I20250905 02:21:06.482065 17756 raft_consensus.cc:2802] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:21:06.482355 17756 raft_consensus.cc:491] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:21:06.482563 17756 raft_consensus.cc:3058] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:06.486811 17756 raft_consensus.cc:513] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.488266 17756 leader_election.cc:290] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [CANDIDATE]: Term 1 election: Requested vote from peers 2a9696d317814d818a702bcf4e131926 (127.15.105.1:39405), 901667520b6d440bb63687299588df60 (127.15.105.3:38045)
I20250905 02:21:06.488886 17419 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db6c9804fe84295835bc5fe09776ec0" candidate_uuid: "430e897ab5954d82b5bd92ef2b1972fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9696d317814d818a702bcf4e131926"
I20250905 02:21:06.489111 17686 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6db6c9804fe84295835bc5fe09776ec0" candidate_uuid: "430e897ab5954d82b5bd92ef2b1972fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "901667520b6d440bb63687299588df60"
I20250905 02:21:06.489295 17419 raft_consensus.cc:3058] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:06.489526 17686 raft_consensus.cc:3058] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:06.493364 17419 raft_consensus.cc:2466] T 6db6c9804fe84295835bc5fe09776ec0 P 2a9696d317814d818a702bcf4e131926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430e897ab5954d82b5bd92ef2b1972fd in term 1.
I20250905 02:21:06.493628 17686 raft_consensus.cc:2466] T 6db6c9804fe84295835bc5fe09776ec0 P 901667520b6d440bb63687299588df60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 430e897ab5954d82b5bd92ef2b1972fd in term 1.
I20250905 02:21:06.494208 17486 leader_election.cc:304] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9696d317814d818a702bcf4e131926, 430e897ab5954d82b5bd92ef2b1972fd; no voters:
I20250905 02:21:06.494778 17756 raft_consensus.cc:2802] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:06.495172 17756 raft_consensus.cc:695] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [term 1 LEADER]: Becoming Leader. State: Replica: 430e897ab5954d82b5bd92ef2b1972fd, State: Running, Role: LEADER
I20250905 02:21:06.495882 17756 consensus_queue.cc:237] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } }
I20250905 02:21:06.502184 17274 catalog_manager.cc:5582] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 430e897ab5954d82b5bd92ef2b1972fd (127.15.105.2). New cstate: current_term: 1 leader_uuid: "430e897ab5954d82b5bd92ef2b1972fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "430e897ab5954d82b5bd92ef2b1972fd" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 35485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 } health_report { overall_health: UNKNOWN } } }
I20250905 02:21:06.701869 17756 consensus_queue.cc:1035] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:21:06.720824 17756 consensus_queue.cc:1035] T 92778bb37ab94ca096dfecdcc7c5e8b2 P 430e897ab5954d82b5bd92ef2b1972fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:21:06.873958 17768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:06.874511 17768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:06.905638 17768 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250905 02:21:06.946187 17756 consensus_queue.cc:1035] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9696d317814d818a702bcf4e131926" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:21:06.976104 17761 consensus_queue.cc:1035] T 6db6c9804fe84295835bc5fe09776ec0 P 430e897ab5954d82b5bd92ef2b1972fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "901667520b6d440bb63687299588df60" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:21:08.229943 17768 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.283s user 0.503s sys 0.775s
W20250905 02:21:08.230363 17768 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.284s user 0.503s sys 0.775s
W20250905 02:21:09.623612 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:09.624246 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:09.657138 17790 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250905 02:21:10.989599 17790 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.292s user 0.455s sys 0.835s
W20250905 02:21:10.990044 17790 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.292s user 0.455s sys 0.835s
W20250905 02:21:12.388953 17805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:12.389500 17805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:12.422633 17805 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250905 02:21:13.728987 17805 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.265s user 0.474s sys 0.790s
W20250905 02:21:13.729359 17805 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.266s user 0.474s sys 0.790s
W20250905 02:21:15.101647 17821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:15.102249 17821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:15.133138 17821 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250905 02:21:16.407931 17821 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.236s user 0.537s sys 0.698s
W20250905 02:21:16.408237 17821 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.236s user 0.537s sys 0.698s
I20250905 02:21:17.487452 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 17335
I20250905 02:21:17.513038 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 17468
I20250905 02:21:17.540915 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 17602
I20250905 02:21:17.565336 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 17243
2025-09-05T02:21:17Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (20732 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250905 02:21:17.625381 15780 test_util.cc:276] Using random seed: 2116570123
I20250905 02:21:17.629454 15780 ts_itest-base.cc:115] Starting cluster with:
I20250905 02:21:17.629607 15780 ts_itest-base.cc:116] --------------
I20250905 02:21:17.629777 15780 ts_itest-base.cc:117] 3 tablet servers
I20250905 02:21:17.629931 15780 ts_itest-base.cc:118] 3 replicas per TS
I20250905 02:21:17.630084 15780 ts_itest-base.cc:119] --------------
2025-09-05T02:21:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T02:21:17Z Disabled control of system clock
I20250905 02:21:17.664867 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:39611
--webserver_interface=127.15.105.62
--webserver_port=0
--builtin_ntp_servers=127.15.105.20:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:39611
--superuser_acl=no-such-user with env {}
W20250905 02:21:17.960054 17845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:17.960593 17845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:17.961051 17845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:17.992486 17845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:21:17.992759 17845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:17.992959 17845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:21:17.993146 17845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:21:18.031883 17845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:39611
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:39611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:18.033082 17845 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:18.034638 17845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:18.045255 17851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:18.046576 17852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:18.048009 17854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:19.265787 17853 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1216 milliseconds
I20250905 02:21:19.265915 17845 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:19.267222 17845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:19.269887 17845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:19.271283 17845 hybrid_clock.cc:648] HybridClock initialized: now 1757038879271238 us; error 52 us; skew 500 ppm
I20250905 02:21:19.272158 17845 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:19.278891 17845 webserver.cc:480] Webserver started at http://127.15.105.62:44915/ using document root <none> and password file <none>
I20250905 02:21:19.279951 17845 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:19.280189 17845 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:19.280650 17845 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:19.285081 17845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4c9050c057184693b0873e07230a3a80"
format_stamp: "Formatted at 2025-09-05 02:21:19 on dist-test-slave-ctw4"
I20250905 02:21:19.286159 17845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4c9050c057184693b0873e07230a3a80"
format_stamp: "Formatted at 2025-09-05 02:21:19 on dist-test-slave-ctw4"
I20250905 02:21:19.293406 17845 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250905 02:21:19.298947 17861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:19.300087 17845 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250905 02:21:19.300443 17845 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4c9050c057184693b0873e07230a3a80"
format_stamp: "Formatted at 2025-09-05 02:21:19 on dist-test-slave-ctw4"
I20250905 02:21:19.300791 17845 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:19.364890 17845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:19.366400 17845 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:19.366853 17845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:19.438195 17845 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:39611
I20250905 02:21:19.438272 17912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:39611 every 8 connection(s)
I20250905 02:21:19.440968 17845 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:21:19.443892 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 17845
I20250905 02:21:19.444415 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 02:21:19.448187 17913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:19.471675 17913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80: Bootstrap starting.
I20250905 02:21:19.477727 17913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:19.479485 17913 log.cc:826] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:19.484134 17913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80: No bootstrap required, opened a new log
I20250905 02:21:19.502590 17913 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9050c057184693b0873e07230a3a80" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39611 } }
I20250905 02:21:19.503273 17913 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:19.503479 17913 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9050c057184693b0873e07230a3a80, State: Initialized, Role: FOLLOWER
I20250905 02:21:19.504130 17913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9050c057184693b0873e07230a3a80" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39611 } }
I20250905 02:21:19.504621 17913 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:19.504858 17913 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:19.505247 17913 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:19.510013 17913 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9050c057184693b0873e07230a3a80" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39611 } }
I20250905 02:21:19.510700 17913 leader_election.cc:304] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c9050c057184693b0873e07230a3a80; no voters:
I20250905 02:21:19.512368 17913 leader_election.cc:290] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:19.513134 17918 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:19.515425 17918 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [term 1 LEADER]: Becoming Leader. State: Replica: 4c9050c057184693b0873e07230a3a80, State: Running, Role: LEADER
I20250905 02:21:19.516268 17918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9050c057184693b0873e07230a3a80" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39611 } }
I20250905 02:21:19.517437 17913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:21:19.529193 17919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c9050c057184693b0873e07230a3a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9050c057184693b0873e07230a3a80" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39611 } } }
I20250905 02:21:19.529103 17920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9050c057184693b0873e07230a3a80. Latest consensus state: current_term: 1 leader_uuid: "4c9050c057184693b0873e07230a3a80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9050c057184693b0873e07230a3a80" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39611 } } }
I20250905 02:21:19.529935 17919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:19.530037 17920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80 [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:19.532537 17928 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:21:19.545020 17928 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:21:19.562222 17928 catalog_manager.cc:1349] Generated new cluster ID: 231fe907edf34a67bb2247af32518e6b
I20250905 02:21:19.562490 17928 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:21:19.590270 17928 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:21:19.591674 17928 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:21:19.608096 17928 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4c9050c057184693b0873e07230a3a80: Generated new TSK 0
I20250905 02:21:19.608985 17928 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:21:19.623162 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:0
--local_ip_for_outbound_sockets=127.15.105.1
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39611
--builtin_ntp_servers=127.15.105.20:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 02:21:19.938444 17937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:19.938961 17937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:19.939455 17937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:19.971474 17937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:19.972429 17937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:21:20.008162 17937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39611
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:20.009526 17937 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:20.011152 17937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:20.031719 17943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:21.436072 17942 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 17937
W20250905 02:21:20.034708 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:21.685810 17942 kernel_stack_watchdog.cc:198] Thread 17937 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:21:21.685909 17937 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.655s user 0.570s sys 1.079s
W20250905 02:21:21.686443 17937 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.655s user 0.571s sys 1.080s
I20250905 02:21:21.694792 17937 server_base.cc:1047] running on GCE node
W20250905 02:21:21.695299 17948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:21.696465 17937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:21.698455 17937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:21.699882 17937 hybrid_clock.cc:648] HybridClock initialized: now 1757038881699831 us; error 49 us; skew 500 ppm
I20250905 02:21:21.700714 17937 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:21.706835 17937 webserver.cc:480] Webserver started at http://127.15.105.1:43083/ using document root <none> and password file <none>
I20250905 02:21:21.707876 17937 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:21.708106 17937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:21.708556 17937 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:21.713109 17937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6cf6c22542d943d69f2afff3bbc68258"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-ctw4"
I20250905 02:21:21.714282 17937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6cf6c22542d943d69f2afff3bbc68258"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-ctw4"
I20250905 02:21:21.721732 17937 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250905 02:21:21.727625 17953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.728785 17937 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250905 02:21:21.729179 17937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6cf6c22542d943d69f2afff3bbc68258"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-ctw4"
I20250905 02:21:21.729540 17937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:21.798286 17937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:21.799852 17937 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:21.800336 17937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:21.802848 17937 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:21.807054 17937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:21.807304 17937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.807580 17937 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:21.807742 17937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.948087 17937 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:42923
I20250905 02:21:21.948201 18065 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:42923 every 8 connection(s)
I20250905 02:21:21.950757 17937 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:21:21.956565 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 17937
I20250905 02:21:21.956986 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 02:21:21.963176 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:0
--local_ip_for_outbound_sockets=127.15.105.2
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39611
--builtin_ntp_servers=127.15.105.20:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:21.978008 18066 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39611
I20250905 02:21:21.978639 18066 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:21.980355 18066 heartbeater.cc:507] Master 127.15.105.62:39611 requested a full tablet report, sending...
I20250905 02:21:21.983705 17878 ts_manager.cc:194] Registered new tserver with Master: 6cf6c22542d943d69f2afff3bbc68258 (127.15.105.1:42923)
I20250905 02:21:21.985723 17878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:33317
W20250905 02:21:22.273097 18070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:22.273627 18070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:22.274140 18070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:22.306947 18070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:22.307874 18070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:21:22.343447 18070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39611
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:22.344849 18070 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:22.347285 18070 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:22.363412 18077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:22.990099 18066 heartbeater.cc:499] Master 127.15.105.62:39611 was elected leader, sending a full tablet report...
W20250905 02:21:23.767220 18076 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 18070
W20250905 02:21:24.131633 18070 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.768s user 0.682s sys 1.072s
W20250905 02:21:22.365808 18078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:24.132069 18070 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.768s user 0.683s sys 1.072s
W20250905 02:21:24.134181 18080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:24.139355 18079 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1774 milliseconds
I20250905 02:21:24.139393 18070 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:24.140643 18070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:24.142717 18070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:24.144104 18070 hybrid_clock.cc:648] HybridClock initialized: now 1757038884144051 us; error 60 us; skew 500 ppm
I20250905 02:21:24.144904 18070 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:24.151019 18070 webserver.cc:480] Webserver started at http://127.15.105.2:33733/ using document root <none> and password file <none>
I20250905 02:21:24.152110 18070 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:24.152356 18070 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:24.152808 18070 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:24.157584 18070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3a94d245275e416ca114d6c80296ce4c"
format_stamp: "Formatted at 2025-09-05 02:21:24 on dist-test-slave-ctw4"
I20250905 02:21:24.158756 18070 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3a94d245275e416ca114d6c80296ce4c"
format_stamp: "Formatted at 2025-09-05 02:21:24 on dist-test-slave-ctw4"
I20250905 02:21:24.166534 18070 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250905 02:21:24.172617 18087 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:24.173748 18070 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250905 02:21:24.174113 18070 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3a94d245275e416ca114d6c80296ce4c"
format_stamp: "Formatted at 2025-09-05 02:21:24 on dist-test-slave-ctw4"
I20250905 02:21:24.174471 18070 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:24.239001 18070 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:24.240656 18070 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:24.241145 18070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:24.244192 18070 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:24.248593 18070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:24.248835 18070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:24.249100 18070 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:24.249274 18070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:24.388792 18070 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:40631
I20250905 02:21:24.388916 18199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:40631 every 8 connection(s)
I20250905 02:21:24.391499 18070 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 02:21:24.392280 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 18070
I20250905 02:21:24.392961 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 02:21:24.402544 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:0
--local_ip_for_outbound_sockets=127.15.105.3
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39611
--builtin_ntp_servers=127.15.105.20:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:24.416348 18200 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39611
I20250905 02:21:24.416796 18200 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:24.417865 18200 heartbeater.cc:507] Master 127.15.105.62:39611 requested a full tablet report, sending...
I20250905 02:21:24.420205 17878 ts_manager.cc:194] Registered new tserver with Master: 3a94d245275e416ca114d6c80296ce4c (127.15.105.2:40631)
I20250905 02:21:24.421438 17878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:59867
W20250905 02:21:24.714061 18204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:24.714601 18204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:24.715108 18204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:24.747577 18204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:24.748503 18204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:21:24.784395 18204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39611
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:24.785771 18204 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:24.787448 18204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:24.799129 18210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:25.425693 18200 heartbeater.cc:499] Master 127.15.105.62:39611 was elected leader, sending a full tablet report...
W20250905 02:21:24.805019 18213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:24.800740 18211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:26.014096 18212 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1213 milliseconds
I20250905 02:21:26.014164 18204 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:26.020473 18204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:26.022645 18204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:26.024037 18204 hybrid_clock.cc:648] HybridClock initialized: now 1757038886024007 us; error 36 us; skew 500 ppm
I20250905 02:21:26.024844 18204 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:26.031580 18204 webserver.cc:480] Webserver started at http://127.15.105.3:39763/ using document root <none> and password file <none>
I20250905 02:21:26.032604 18204 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:26.032850 18204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:26.033349 18204 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:26.038101 18204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "73cd844327f645dcaa6cbdd52e32a128"
format_stamp: "Formatted at 2025-09-05 02:21:26 on dist-test-slave-ctw4"
I20250905 02:21:26.039294 18204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "73cd844327f645dcaa6cbdd52e32a128"
format_stamp: "Formatted at 2025-09-05 02:21:26 on dist-test-slave-ctw4"
I20250905 02:21:26.046653 18204 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20250905 02:21:26.052660 18220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:26.053748 18204 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250905 02:21:26.054126 18204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "73cd844327f645dcaa6cbdd52e32a128"
format_stamp: "Formatted at 2025-09-05 02:21:26 on dist-test-slave-ctw4"
I20250905 02:21:26.054487 18204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:26.107554 18204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:26.109138 18204 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:26.109743 18204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:26.112563 18204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:26.116880 18204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:26.117120 18204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:26.117327 18204 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:26.117460 18204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:26.257117 18204 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:32869
I20250905 02:21:26.257225 18332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:32869 every 8 connection(s)
I20250905 02:21:26.259809 18204 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 02:21:26.262384 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 18204
I20250905 02:21:26.262961 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 02:21:26.281740 18333 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39611
I20250905 02:21:26.282266 18333 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:26.283488 18333 heartbeater.cc:507] Master 127.15.105.62:39611 requested a full tablet report, sending...
I20250905 02:21:26.285743 17878 ts_manager.cc:194] Registered new tserver with Master: 73cd844327f645dcaa6cbdd52e32a128 (127.15.105.3:32869)
I20250905 02:21:26.287041 17878 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:34071
I20250905 02:21:26.298586 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:21:26.334491 17878 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52930:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250905 02:21:26.354779 17878 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:21:26.423593 18268 tablet_service.cc:1468] Processing CreateTablet for tablet 5bc2e269a64b4bfd9fcde5d46e33fb36 (DEFAULT_TABLE table=TestTable [id=e930c49a21fa48138eaacb0e9474965f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:26.424358 18001 tablet_service.cc:1468] Processing CreateTablet for tablet 5bc2e269a64b4bfd9fcde5d46e33fb36 (DEFAULT_TABLE table=TestTable [id=e930c49a21fa48138eaacb0e9474965f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:26.425403 18135 tablet_service.cc:1468] Processing CreateTablet for tablet 5bc2e269a64b4bfd9fcde5d46e33fb36 (DEFAULT_TABLE table=TestTable [id=e930c49a21fa48138eaacb0e9474965f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:26.426026 18268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc2e269a64b4bfd9fcde5d46e33fb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:26.426249 18001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc2e269a64b4bfd9fcde5d46e33fb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:26.427081 18135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc2e269a64b4bfd9fcde5d46e33fb36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:26.463117 18352 tablet_bootstrap.cc:492] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128: Bootstrap starting.
I20250905 02:21:26.473237 18354 tablet_bootstrap.cc:492] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258: Bootstrap starting.
I20250905 02:21:26.475448 18352 tablet_bootstrap.cc:654] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:26.477128 18353 tablet_bootstrap.cc:492] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c: Bootstrap starting.
I20250905 02:21:26.477993 18352 log.cc:826] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:26.484467 18353 tablet_bootstrap.cc:654] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:26.486740 18353 log.cc:826] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:26.493605 18354 tablet_bootstrap.cc:654] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:26.496606 18354 log.cc:826] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:26.503712 18352 tablet_bootstrap.cc:492] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128: No bootstrap required, opened a new log
I20250905 02:21:26.504442 18352 ts_tablet_manager.cc:1397] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128: Time spent bootstrapping tablet: real 0.044s user 0.027s sys 0.011s
I20250905 02:21:26.507509 18353 tablet_bootstrap.cc:492] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c: No bootstrap required, opened a new log
I20250905 02:21:26.507723 18354 tablet_bootstrap.cc:492] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258: No bootstrap required, opened a new log
I20250905 02:21:26.508065 18353 ts_tablet_manager.cc:1397] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.015s
I20250905 02:21:26.508283 18354 ts_tablet_manager.cc:1397] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.008s
I20250905 02:21:26.530408 18352 raft_consensus.cc:357] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.531108 18352 raft_consensus.cc:383] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:26.531387 18352 raft_consensus.cc:738] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cd844327f645dcaa6cbdd52e32a128, State: Initialized, Role: FOLLOWER
I20250905 02:21:26.532188 18352 consensus_queue.cc:260] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.536016 18333 heartbeater.cc:499] Master 127.15.105.62:39611 was elected leader, sending a full tablet report...
I20250905 02:21:26.536613 18352 ts_tablet_manager.cc:1428] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128: Time spent starting tablet: real 0.032s user 0.026s sys 0.005s
I20250905 02:21:26.536561 18354 raft_consensus.cc:357] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.537425 18354 raft_consensus.cc:383] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:26.537716 18354 raft_consensus.cc:738] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf6c22542d943d69f2afff3bbc68258, State: Initialized, Role: FOLLOWER
I20250905 02:21:26.538595 18354 consensus_queue.cc:260] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.539145 18353 raft_consensus.cc:357] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.540717 18353 raft_consensus.cc:383] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:26.541190 18353 raft_consensus.cc:738] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a94d245275e416ca114d6c80296ce4c, State: Initialized, Role: FOLLOWER
I20250905 02:21:26.542554 18353 consensus_queue.cc:260] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.544516 18354 ts_tablet_manager.cc:1428] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258: Time spent starting tablet: real 0.036s user 0.026s sys 0.008s
I20250905 02:21:26.549273 18353 ts_tablet_manager.cc:1428] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c: Time spent starting tablet: real 0.041s user 0.030s sys 0.002s
W20250905 02:21:26.649344 18201 tablet.cc:2378] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:21:26.695941 18358 raft_consensus.cc:491] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:21:26.696364 18358 raft_consensus.cc:513] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.698540 18358 leader_election.cc:290] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cf6c22542d943d69f2afff3bbc68258 (127.15.105.1:42923), 3a94d245275e416ca114d6c80296ce4c (127.15.105.2:40631)
I20250905 02:21:26.707520 18021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc2e269a64b4bfd9fcde5d46e33fb36" candidate_uuid: "73cd844327f645dcaa6cbdd52e32a128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf6c22542d943d69f2afff3bbc68258" is_pre_election: true
I20250905 02:21:26.708392 18021 raft_consensus.cc:2466] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73cd844327f645dcaa6cbdd52e32a128 in term 0.
I20250905 02:21:26.709123 18155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc2e269a64b4bfd9fcde5d46e33fb36" candidate_uuid: "73cd844327f645dcaa6cbdd52e32a128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94d245275e416ca114d6c80296ce4c" is_pre_election: true
I20250905 02:21:26.709597 18221 leader_election.cc:304] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf6c22542d943d69f2afff3bbc68258, 73cd844327f645dcaa6cbdd52e32a128; no voters:
I20250905 02:21:26.709781 18155 raft_consensus.cc:2466] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73cd844327f645dcaa6cbdd52e32a128 in term 0.
I20250905 02:21:26.710345 18358 raft_consensus.cc:2802] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:21:26.710602 18358 raft_consensus.cc:491] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:21:26.710866 18358 raft_consensus.cc:3058] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 0 FOLLOWER]: Advancing to term 1
W20250905 02:21:26.711287 18067 tablet.cc:2378] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:21:26.716414 18358 raft_consensus.cc:513] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.717730 18358 leader_election.cc:290] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [CANDIDATE]: Term 1 election: Requested vote from peers 6cf6c22542d943d69f2afff3bbc68258 (127.15.105.1:42923), 3a94d245275e416ca114d6c80296ce4c (127.15.105.2:40631)
I20250905 02:21:26.718372 18021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc2e269a64b4bfd9fcde5d46e33fb36" candidate_uuid: "73cd844327f645dcaa6cbdd52e32a128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf6c22542d943d69f2afff3bbc68258"
I20250905 02:21:26.718617 18155 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bc2e269a64b4bfd9fcde5d46e33fb36" candidate_uuid: "73cd844327f645dcaa6cbdd52e32a128" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a94d245275e416ca114d6c80296ce4c"
I20250905 02:21:26.718772 18021 raft_consensus.cc:3058] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:26.719103 18155 raft_consensus.cc:3058] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:26.723172 18021 raft_consensus.cc:2466] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 6cf6c22542d943d69f2afff3bbc68258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73cd844327f645dcaa6cbdd52e32a128 in term 1.
I20250905 02:21:26.723332 18155 raft_consensus.cc:2466] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 3a94d245275e416ca114d6c80296ce4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73cd844327f645dcaa6cbdd52e32a128 in term 1.
I20250905 02:21:26.724011 18221 leader_election.cc:304] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf6c22542d943d69f2afff3bbc68258, 73cd844327f645dcaa6cbdd52e32a128; no voters:
I20250905 02:21:26.724732 18358 raft_consensus.cc:2802] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:26.726272 18358 raft_consensus.cc:695] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [term 1 LEADER]: Becoming Leader. State: Replica: 73cd844327f645dcaa6cbdd52e32a128, State: Running, Role: LEADER
I20250905 02:21:26.727154 18358 consensus_queue.cc:237] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } }
I20250905 02:21:26.737068 17878 catalog_manager.cc:5582] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73cd844327f645dcaa6cbdd52e32a128 (127.15.105.3). New cstate: current_term: 1 leader_uuid: "73cd844327f645dcaa6cbdd52e32a128" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cd844327f645dcaa6cbdd52e32a128" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 32869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 } health_report { overall_health: UNKNOWN } } }
W20250905 02:21:26.764657 18334 tablet.cc:2378] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:21:26.796134 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:21:26.799571 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6cf6c22542d943d69f2afff3bbc68258 to finish bootstrapping
I20250905 02:21:26.813535 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3a94d245275e416ca114d6c80296ce4c to finish bootstrapping
I20250905 02:21:26.824570 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 73cd844327f645dcaa6cbdd52e32a128 to finish bootstrapping
I20250905 02:21:27.307613 18375 consensus_queue.cc:1035] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a94d245275e416ca114d6c80296ce4c" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:21:27.320801 18375 consensus_queue.cc:1035] T 5bc2e269a64b4bfd9fcde5d46e33fb36 P 73cd844327f645dcaa6cbdd52e32a128 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf6c22542d943d69f2afff3bbc68258" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 42923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 02:21:28.558804 17878 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:35368
I20250905 02:21:29.588668 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 17937
I20250905 02:21:29.612627 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 18070
I20250905 02:21:29.637768 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 18204
I20250905 02:21:29.661016 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 17845
2025-09-05T02:21:29Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (12087 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250905 02:21:29.713150 15780 test_util.cc:276] Using random seed: 2128657889
I20250905 02:21:29.717101 15780 ts_itest-base.cc:115] Starting cluster with:
I20250905 02:21:29.717263 15780 ts_itest-base.cc:116] --------------
I20250905 02:21:29.717414 15780 ts_itest-base.cc:117] 3 tablet servers
I20250905 02:21:29.717553 15780 ts_itest-base.cc:118] 3 replicas per TS
I20250905 02:21:29.717693 15780 ts_itest-base.cc:119] --------------
2025-09-05T02:21:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T02:21:29Z Disabled control of system clock
I20250905 02:21:29.753878 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:41029
--webserver_interface=127.15.105.62
--webserver_port=0
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:41029 with env {}
W20250905 02:21:30.053385 18401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:30.053988 18401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:30.054468 18401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:30.085916 18401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:21:30.086311 18401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:30.086576 18401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:21:30.086822 18401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:21:30.122654 18401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:41029
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:30.124011 18401 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:30.125705 18401 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:30.137614 18407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:30.138707 18408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:31.340737 18410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:31.343583 18409 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1200 milliseconds
I20250905 02:21:31.343747 18401 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:31.345052 18401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:31.347613 18401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:31.349042 18401 hybrid_clock.cc:648] HybridClock initialized: now 1757038891348984 us; error 76 us; skew 500 ppm
I20250905 02:21:31.349900 18401 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:31.356863 18401 webserver.cc:480] Webserver started at http://127.15.105.62:41509/ using document root <none> and password file <none>
I20250905 02:21:31.357898 18401 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:31.358116 18401 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:31.358597 18401 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:31.364240 18401 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5b4815f1687a4367b7328097a0d48ada"
format_stamp: "Formatted at 2025-09-05 02:21:31 on dist-test-slave-ctw4"
I20250905 02:21:31.365444 18401 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5b4815f1687a4367b7328097a0d48ada"
format_stamp: "Formatted at 2025-09-05 02:21:31 on dist-test-slave-ctw4"
I20250905 02:21:31.373330 18401 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250905 02:21:31.379210 18417 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:31.380460 18401 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250905 02:21:31.380888 18401 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5b4815f1687a4367b7328097a0d48ada"
format_stamp: "Formatted at 2025-09-05 02:21:31 on dist-test-slave-ctw4"
I20250905 02:21:31.381259 18401 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:31.458544 18401 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:31.460121 18401 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:31.460579 18401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:31.533128 18401 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:41029
I20250905 02:21:31.533205 18468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:41029 every 8 connection(s)
I20250905 02:21:31.536012 18401 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:21:31.541916 18469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:31.544862 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 18401
I20250905 02:21:31.545292 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 02:21:31.567185 18469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap starting.
I20250905 02:21:31.574558 18469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:31.576472 18469 log.cc:826] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:31.581504 18469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: No bootstrap required, opened a new log
I20250905 02:21:31.601230 18469 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:31.601886 18469 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:31.602087 18469 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4815f1687a4367b7328097a0d48ada, State: Initialized, Role: FOLLOWER
I20250905 02:21:31.602713 18469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:31.603211 18469 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:31.603449 18469 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:31.603708 18469 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:31.608729 18469 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:31.609499 18469 leader_election.cc:304] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4815f1687a4367b7328097a0d48ada; no voters:
I20250905 02:21:31.611179 18469 leader_election.cc:290] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:31.612030 18474 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:31.614296 18474 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 1 LEADER]: Becoming Leader. State: Replica: 5b4815f1687a4367b7328097a0d48ada, State: Running, Role: LEADER
I20250905 02:21:31.615187 18474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:31.615470 18469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:21:31.626919 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b4815f1687a4367b7328097a0d48ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } } }
I20250905 02:21:31.628357 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:31.628626 18476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4815f1687a4367b7328097a0d48ada. Latest consensus state: current_term: 1 leader_uuid: "5b4815f1687a4367b7328097a0d48ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } } }
I20250905 02:21:31.629609 18476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:31.639623 18483 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:21:31.651743 18483 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:21:31.669137 18483 catalog_manager.cc:1349] Generated new cluster ID: dc398bad0ff543d888d4b46d29e02845
I20250905 02:21:31.669457 18483 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:21:31.700443 18483 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:21:31.701831 18483 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:21:31.717114 18483 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Generated new TSK 0
I20250905 02:21:31.717978 18483 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:21:31.745575 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:0
--local_ip_for_outbound_sockets=127.15.105.1
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41029
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 02:21:32.054374 18493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:32.054925 18493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:32.055438 18493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:32.088020 18493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:32.088891 18493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:21:32.125141 18493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:32.126520 18493 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:32.128285 18493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:32.142978 18500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.544241 18499 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 18493
W20250905 02:21:32.143687 18501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.724489 18499 kernel_stack_watchdog.cc:198] Thread 18493 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:21:33.724936 18493 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.583s user 0.607s sys 0.975s
W20250905 02:21:33.725309 18493 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.584s user 0.607s sys 0.975s
I20250905 02:21:33.733769 18493 server_base.cc:1047] running on GCE node
W20250905 02:21:33.733958 18505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:33.735157 18493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:33.737278 18493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:33.738649 18493 hybrid_clock.cc:648] HybridClock initialized: now 1757038893738604 us; error 51 us; skew 500 ppm
I20250905 02:21:33.739465 18493 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:33.745625 18493 webserver.cc:480] Webserver started at http://127.15.105.1:32773/ using document root <none> and password file <none>
I20250905 02:21:33.746599 18493 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:33.746835 18493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:33.747280 18493 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:33.751842 18493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6850db245d904b3e977ffc92f91c5c4d"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-ctw4"
I20250905 02:21:33.753031 18493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6850db245d904b3e977ffc92f91c5c4d"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-ctw4"
I20250905 02:21:33.761054 18493 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20250905 02:21:33.767316 18510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.768486 18493 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250905 02:21:33.768844 18493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6850db245d904b3e977ffc92f91c5c4d"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-ctw4"
I20250905 02:21:33.769213 18493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:33.820443 18493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:33.822041 18493 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:33.822520 18493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:33.825127 18493 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:33.829406 18493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:33.829645 18493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.829954 18493 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:33.830116 18493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.968572 18493 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:39371
I20250905 02:21:33.968683 18622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:39371 every 8 connection(s)
I20250905 02:21:33.971236 18493 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:21:33.977770 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 18493
I20250905 02:21:33.978469 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 02:21:33.985738 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:0
--local_ip_for_outbound_sockets=127.15.105.2
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41029
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:33.999094 18623 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:21:33.999563 18623 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:34.000669 18623 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:21:34.003571 18434 ts_manager.cc:194] Registered new tserver with Master: 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1:39371)
I20250905 02:21:34.005769 18434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:43875
W20250905 02:21:34.297284 18627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:34.297848 18627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:34.298364 18627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:34.330122 18627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:34.331015 18627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:21:34.367092 18627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:34.368477 18627 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:34.370133 18627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:34.382066 18633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:35.010188 18623 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
W20250905 02:21:34.382891 18634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:35.603526 18635 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1218 milliseconds
W20250905 02:21:35.604413 18636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:35.607594 18627 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.225s user 0.421s sys 0.804s
W20250905 02:21:35.607877 18627 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.226s user 0.422s sys 0.804s
I20250905 02:21:35.608090 18627 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:35.609102 18627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:35.611335 18627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:35.612673 18627 hybrid_clock.cc:648] HybridClock initialized: now 1757038895612644 us; error 44 us; skew 500 ppm
I20250905 02:21:35.613468 18627 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:35.620558 18627 webserver.cc:480] Webserver started at http://127.15.105.2:40361/ using document root <none> and password file <none>
I20250905 02:21:35.621546 18627 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:35.621789 18627 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:35.622282 18627 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:35.626585 18627 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ec017dec7c9b42128e05beaa6f879077"
format_stamp: "Formatted at 2025-09-05 02:21:35 on dist-test-slave-ctw4"
I20250905 02:21:35.627625 18627 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ec017dec7c9b42128e05beaa6f879077"
format_stamp: "Formatted at 2025-09-05 02:21:35 on dist-test-slave-ctw4"
I20250905 02:21:35.635025 18627 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20250905 02:21:35.640884 18643 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:35.642230 18627 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250905 02:21:35.642563 18627 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ec017dec7c9b42128e05beaa6f879077"
format_stamp: "Formatted at 2025-09-05 02:21:35 on dist-test-slave-ctw4"
I20250905 02:21:35.642874 18627 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:35.709311 18627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:35.710728 18627 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:35.711151 18627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:35.713573 18627 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:35.717741 18627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:35.717960 18627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:35.718190 18627 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:35.718344 18627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:35.846220 18627 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:45857
I20250905 02:21:35.846312 18755 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:45857 every 8 connection(s)
I20250905 02:21:35.848654 18627 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 02:21:35.849866 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 18627
I20250905 02:21:35.850278 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 02:21:35.857244 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:0
--local_ip_for_outbound_sockets=127.15.105.3
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41029
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:35.870460 18756 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:21:35.870870 18756 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:35.871878 18756 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:21:35.873952 18434 ts_manager.cc:194] Registered new tserver with Master: ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857)
I20250905 02:21:35.875141 18434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:50121
W20250905 02:21:36.151432 18760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:36.151969 18760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:36.152453 18760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:36.183458 18760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:36.184314 18760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:21:36.218161 18760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:36.219388 18760 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:36.220913 18760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:36.232882 18766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:36.878090 18756 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
W20250905 02:21:36.233541 18767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:37.431855 18768 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1196 milliseconds
W20250905 02:21:37.432569 18769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:37.435734 18760 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.203s user 0.373s sys 0.816s
W20250905 02:21:37.436053 18760 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.203s user 0.373s sys 0.817s
I20250905 02:21:37.436276 18760 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:37.437256 18760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:37.439390 18760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:37.440718 18760 hybrid_clock.cc:648] HybridClock initialized: now 1757038897440695 us; error 33 us; skew 500 ppm
I20250905 02:21:37.441416 18760 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:37.448042 18760 webserver.cc:480] Webserver started at http://127.15.105.3:46845/ using document root <none> and password file <none>
I20250905 02:21:37.449029 18760 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:37.449265 18760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:37.449735 18760 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:37.454094 18760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2057551e7a3b4d259fc6f55d74fcdbac"
format_stamp: "Formatted at 2025-09-05 02:21:37 on dist-test-slave-ctw4"
I20250905 02:21:37.455111 18760 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2057551e7a3b4d259fc6f55d74fcdbac"
format_stamp: "Formatted at 2025-09-05 02:21:37 on dist-test-slave-ctw4"
I20250905 02:21:37.462167 18760 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250905 02:21:37.468207 18776 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:37.469417 18760 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250905 02:21:37.469708 18760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2057551e7a3b4d259fc6f55d74fcdbac"
format_stamp: "Formatted at 2025-09-05 02:21:37 on dist-test-slave-ctw4"
I20250905 02:21:37.470053 18760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:37.532302 18760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:37.534287 18760 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:37.534844 18760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:37.538044 18760 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:37.543515 18760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:37.543833 18760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250905 02:21:37.544131 18760 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:37.544329 18760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:37.678074 18760 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:37697
I20250905 02:21:37.678182 18888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:37697 every 8 connection(s)
I20250905 02:21:37.680526 18760 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 02:21:37.688985 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 18760
I20250905 02:21:37.689363 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 02:21:37.699847 18889 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:21:37.700251 18889 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:37.701201 18889 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:21:37.703194 18434 ts_manager.cc:194] Registered new tserver with Master: 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697)
I20250905 02:21:37.704396 18434 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:58867
I20250905 02:21:37.708287 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:21:37.742894 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:21:37.743278 15780 test_util.cc:276] Using random seed: 2136688024
I20250905 02:21:37.782886 18434 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48686:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 02:21:37.823221 18824 tablet_service.cc:1468] Processing CreateTablet for tablet 3292527771b441a49fce17c4d90f3a92 (DEFAULT_TABLE table=TestTable [id=f6395f2ed0db4584b30df7d291f43d57]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:37.824705 18824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3292527771b441a49fce17c4d90f3a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:37.843717 18909 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap starting.
I20250905 02:21:37.849241 18909 tablet_bootstrap.cc:654] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:37.851236 18909 log.cc:826] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:37.856401 18909 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: No bootstrap required, opened a new log
I20250905 02:21:37.856801 18909 ts_tablet_manager.cc:1397] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20250905 02:21:37.874011 18909 raft_consensus.cc:357] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } }
I20250905 02:21:37.874567 18909 raft_consensus.cc:383] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:37.874801 18909 raft_consensus.cc:738] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057551e7a3b4d259fc6f55d74fcdbac, State: Initialized, Role: FOLLOWER
I20250905 02:21:37.875484 18909 consensus_queue.cc:260] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } }
I20250905 02:21:37.876122 18909 raft_consensus.cc:397] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:37.876390 18909 raft_consensus.cc:491] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:37.876713 18909 raft_consensus.cc:3058] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:37.881006 18909 raft_consensus.cc:513] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } }
I20250905 02:21:37.881660 18909 leader_election.cc:304] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2057551e7a3b4d259fc6f55d74fcdbac; no voters:
I20250905 02:21:37.883235 18909 leader_election.cc:290] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:37.883625 18911 raft_consensus.cc:2802] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:37.886479 18889 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
I20250905 02:21:37.887192 18911 raft_consensus.cc:695] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 1 LEADER]: Becoming Leader. State: Replica: 2057551e7a3b4d259fc6f55d74fcdbac, State: Running, Role: LEADER
I20250905 02:21:37.887439 18909 ts_tablet_manager.cc:1428] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent starting tablet: real 0.030s user 0.026s sys 0.004s
I20250905 02:21:37.887948 18911 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } }
I20250905 02:21:37.897961 18434 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac reported cstate change: term changed from 0 to 1, leader changed from <none> to 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3). New cstate: current_term: 1 leader_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } health_report { overall_health: HEALTHY } } }
I20250905 02:21:38.092979 15780 test_util.cc:276] Using random seed: 2137037713
I20250905 02:21:38.112928 18431 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48692:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 02:21:38.141399 18558 tablet_service.cc:1468] Processing CreateTablet for tablet b8f259ed87244e51b1e3c174e81cea31 (DEFAULT_TABLE table=TestTable1 [id=f4afdddda4814133acad03e45c43faab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:38.142869 18558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f259ed87244e51b1e3c174e81cea31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:38.161005 18930 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap starting.
I20250905 02:21:38.166635 18930 tablet_bootstrap.cc:654] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:38.168372 18930 log.cc:826] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:38.172771 18930 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: No bootstrap required, opened a new log
I20250905 02:21:38.173206 18930 ts_tablet_manager.cc:1397] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20250905 02:21:38.190510 18930 raft_consensus.cc:357] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } }
I20250905 02:21:38.191255 18930 raft_consensus.cc:383] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:38.191461 18930 raft_consensus.cc:738] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Initialized, Role: FOLLOWER
I20250905 02:21:38.192114 18930 consensus_queue.cc:260] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } }
I20250905 02:21:38.192585 18930 raft_consensus.cc:397] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:38.192813 18930 raft_consensus.cc:491] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:38.193070 18930 raft_consensus.cc:3058] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:38.197212 18930 raft_consensus.cc:513] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } }
I20250905 02:21:38.197844 18930 leader_election.cc:304] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6850db245d904b3e977ffc92f91c5c4d; no voters:
I20250905 02:21:38.199831 18930 leader_election.cc:290] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:38.200255 18932 raft_consensus.cc:2802] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:38.202778 18932 raft_consensus.cc:695] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 1 LEADER]: Becoming Leader. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Running, Role: LEADER
I20250905 02:21:38.203960 18930 ts_tablet_manager.cc:1428] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Time spent starting tablet: real 0.030s user 0.027s sys 0.005s
I20250905 02:21:38.203868 18932 consensus_queue.cc:237] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } }
I20250905 02:21:38.214056 18431 catalog_manager.cc:5582] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1). New cstate: current_term: 1 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } health_report { overall_health: HEALTHY } } }
I20250905 02:21:38.394248 15780 test_util.cc:276] Using random seed: 2137338988
I20250905 02:21:38.415315 18433 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48696:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 02:21:38.444026 18691 tablet_service.cc:1468] Processing CreateTablet for tablet b433bc2d1fc847dbb0e072e941f0f0cb (DEFAULT_TABLE table=TestTable2 [id=194eb01f9c6249879c384f0c773ebc97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:38.445616 18691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b433bc2d1fc847dbb0e072e941f0f0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:38.464498 18951 tablet_bootstrap.cc:492] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Bootstrap starting.
I20250905 02:21:38.470327 18951 tablet_bootstrap.cc:654] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:38.472076 18951 log.cc:826] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:38.476485 18951 tablet_bootstrap.cc:492] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: No bootstrap required, opened a new log
I20250905 02:21:38.476974 18951 ts_tablet_manager.cc:1397] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20250905 02:21:38.497136 18951 raft_consensus.cc:357] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:21:38.497670 18951 raft_consensus.cc:383] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:38.497874 18951 raft_consensus.cc:738] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Initialized, Role: FOLLOWER
I20250905 02:21:38.498483 18951 consensus_queue.cc:260] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:21:38.498951 18951 raft_consensus.cc:397] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:38.499192 18951 raft_consensus.cc:491] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:38.499442 18951 raft_consensus.cc:3058] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:38.503563 18951 raft_consensus.cc:513] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:21:38.504273 18951 leader_election.cc:304] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec017dec7c9b42128e05beaa6f879077; no voters:
I20250905 02:21:38.505884 18951 leader_election.cc:290] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:38.506274 18953 raft_consensus.cc:2802] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:38.508337 18953 raft_consensus.cc:695] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 1 LEADER]: Becoming Leader. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Running, Role: LEADER
I20250905 02:21:38.509321 18953 consensus_queue.cc:237] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:21:38.509738 18951 ts_tablet_manager.cc:1428] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Time spent starting tablet: real 0.032s user 0.026s sys 0.008s
I20250905 02:21:38.517866 18433 catalog_manager.cc:5582] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec017dec7c9b42128e05beaa6f879077 (127.15.105.2). New cstate: current_term: 1 leader_uuid: "ec017dec7c9b42128e05beaa6f879077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } health_report { overall_health: HEALTHY } } }
I20250905 02:21:38.687394 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 18401
W20250905 02:21:38.912253 18889 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:41029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:41029: connect: Connection refused (error 111)
W20250905 02:21:39.229076 18623 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:41029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:41029: connect: Connection refused (error 111)
W20250905 02:21:39.534389 18756 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:41029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:41029: connect: Connection refused (error 111)
I20250905 02:21:43.644337 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 18493
I20250905 02:21:43.666920 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 18627
I20250905 02:21:43.701247 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 18760
I20250905 02:21:43.729112 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:41029
--webserver_interface=127.15.105.62
--webserver_port=41509
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:41029 with env {}
W20250905 02:21:44.031545 19031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:44.032169 19031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:44.032609 19031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:44.063000 19031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:21:44.063354 19031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:44.063607 19031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:21:44.063870 19031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:21:44.099020 19031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:41029
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=41509
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:44.100351 19031 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:44.101946 19031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:44.112929 19037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:44.113507 19038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:45.309718 19040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:45.312075 19039 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1194 milliseconds
I20250905 02:21:45.312171 19031 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:45.313359 19031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:45.315852 19031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:45.317205 19031 hybrid_clock.cc:648] HybridClock initialized: now 1757038905317162 us; error 44 us; skew 500 ppm
I20250905 02:21:45.317981 19031 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:45.323889 19031 webserver.cc:480] Webserver started at http://127.15.105.62:41509/ using document root <none> and password file <none>
I20250905 02:21:45.324761 19031 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:45.324988 19031 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:45.332407 19031 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250905 02:21:45.336807 19047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:45.337751 19031 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250905 02:21:45.338068 19031 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5b4815f1687a4367b7328097a0d48ada"
format_stamp: "Formatted at 2025-09-05 02:21:31 on dist-test-slave-ctw4"
I20250905 02:21:45.339987 19031 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:45.392158 19031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:45.393585 19031 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:45.394032 19031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:45.462247 19031 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:41029
I20250905 02:21:45.462327 19098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:41029 every 8 connection(s)
I20250905 02:21:45.464980 19031 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:21:45.469844 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 19031
I20250905 02:21:45.471225 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:39371
--local_ip_for_outbound_sockets=127.15.105.1
--tserver_master_addrs=127.15.105.62:41029
--webserver_port=32773
--webserver_interface=127.15.105.1
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:45.475438 19099 sys_catalog.cc:263] Verifying existing consensus state
I20250905 02:21:45.481513 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap starting.
I20250905 02:21:45.496551 19099 log.cc:826] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:45.556689 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:45.557487 19099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap complete.
I20250905 02:21:45.578389 19099 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:45.580471 19099 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4815f1687a4367b7328097a0d48ada, State: Initialized, Role: FOLLOWER
I20250905 02:21:45.581215 19099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:45.581759 19099 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:45.582012 19099 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:45.582302 19099 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:21:45.587539 19099 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:45.588145 19099 leader_election.cc:304] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4815f1687a4367b7328097a0d48ada; no voters:
I20250905 02:21:45.590204 19099 leader_election.cc:290] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [CANDIDATE]: Term 3 election: Requested vote from peers
I20250905 02:21:45.590590 19103 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:21:45.593920 19103 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 3 LEADER]: Becoming Leader. State: Replica: 5b4815f1687a4367b7328097a0d48ada, State: Running, Role: LEADER
I20250905 02:21:45.594763 19103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:45.595130 19099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:21:45.606303 19104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5b4815f1687a4367b7328097a0d48ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } } }
I20250905 02:21:45.607038 19104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:45.608644 19105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4815f1687a4367b7328097a0d48ada. Latest consensus state: current_term: 3 leader_uuid: "5b4815f1687a4367b7328097a0d48ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } } }
I20250905 02:21:45.610852 19105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:45.612850 19111 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:21:45.626670 19111 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=194eb01f9c6249879c384f0c773ebc97]
I20250905 02:21:45.628429 19111 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=c2ff4768efa3495b809568d02a82ccae]
I20250905 02:21:45.630123 19111 catalog_manager.cc:671] Loaded metadata for table TestTable [id=c45a8adf354444c4991e4809f345da7e]
I20250905 02:21:45.637902 19111 tablet_loader.cc:96] loaded metadata for tablet 3292527771b441a49fce17c4d90f3a92 (table TestTable [id=c45a8adf354444c4991e4809f345da7e])
I20250905 02:21:45.639195 19111 tablet_loader.cc:96] loaded metadata for tablet b433bc2d1fc847dbb0e072e941f0f0cb (table TestTable2 [id=194eb01f9c6249879c384f0c773ebc97])
I20250905 02:21:45.640421 19111 tablet_loader.cc:96] loaded metadata for tablet b8f259ed87244e51b1e3c174e81cea31 (table TestTable1 [id=c2ff4768efa3495b809568d02a82ccae])
I20250905 02:21:45.641785 19111 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:21:45.647794 19111 catalog_manager.cc:1261] Loaded cluster ID: dc398bad0ff543d888d4b46d29e02845
I20250905 02:21:45.648119 19111 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:21:45.656111 19111 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:21:45.661583 19111 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Loaded TSK: 0
I20250905 02:21:45.663206 19111 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250905 02:21:45.824307 19101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:45.824800 19101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:45.825292 19101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:45.856099 19101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:45.857026 19101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:21:45.892073 19101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:39371
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=32773
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:45.893357 19101 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:45.894968 19101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:45.907058 19126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:47.310190 19125 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 19101
W20250905 02:21:47.425083 19125 kernel_stack_watchdog.cc:198] Thread 19101 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:21:45.908226 19127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:47.426997 19101 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.519s user 0.532s sys 0.972s
W20250905 02:21:47.427464 19101 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.520s user 0.532s sys 0.972s
W20250905 02:21:47.428683 19129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:47.431080 19128 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1519 milliseconds
I20250905 02:21:47.431145 19101 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:47.432282 19101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:47.434716 19101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:47.436142 19101 hybrid_clock.cc:648] HybridClock initialized: now 1757038907436087 us; error 57 us; skew 500 ppm
I20250905 02:21:47.436875 19101 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:47.443147 19101 webserver.cc:480] Webserver started at http://127.15.105.1:32773/ using document root <none> and password file <none>
I20250905 02:21:47.444154 19101 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:47.444386 19101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:47.452406 19101 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.002s
I20250905 02:21:47.458217 19136 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:47.459308 19101 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250905 02:21:47.459614 19101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6850db245d904b3e977ffc92f91c5c4d"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-ctw4"
I20250905 02:21:47.461642 19101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:47.510042 19101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:47.511452 19101 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:47.511919 19101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:47.515059 19101 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:47.521147 19143 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 02:21:47.529003 19101 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 02:21:47.529217 19101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250905 02:21:47.529505 19101 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 02:21:47.533936 19101 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 02:21:47.534190 19101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.001s sys 0.002s
I20250905 02:21:47.534540 19143 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap starting.
I20250905 02:21:47.614967 19143 log.cc:826] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:47.733108 19143 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:47.734088 19143 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap complete.
I20250905 02:21:47.735936 19143 ts_tablet_manager.cc:1397] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Time spent bootstrapping tablet: real 0.202s user 0.156s sys 0.036s
I20250905 02:21:47.739630 19101 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:39371
I20250905 02:21:47.739830 19250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:39371 every 8 connection(s)
I20250905 02:21:47.742122 19101 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:21:47.745857 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 19101
I20250905 02:21:47.747637 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:45857
--local_ip_for_outbound_sockets=127.15.105.2
--tserver_master_addrs=127.15.105.62:41029
--webserver_port=40361
--webserver_interface=127.15.105.2
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:47.767002 19143 raft_consensus.cc:357] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } }
I20250905 02:21:47.769999 19143 raft_consensus.cc:738] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Initialized, Role: FOLLOWER
I20250905 02:21:47.770912 19143 consensus_queue.cc:260] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } }
I20250905 02:21:47.771690 19143 raft_consensus.cc:397] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:47.772080 19143 raft_consensus.cc:491] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:47.772485 19143 raft_consensus.cc:3058] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:21:47.779492 19143 raft_consensus.cc:513] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } }
I20250905 02:21:47.780134 19143 leader_election.cc:304] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6850db245d904b3e977ffc92f91c5c4d; no voters:
I20250905 02:21:47.789933 19251 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:21:47.790551 19251 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:47.792121 19251 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:21:47.792368 19143 leader_election.cc:290] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 02:21:47.792850 19256 raft_consensus.cc:2802] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:21:47.797897 19256 raft_consensus.cc:695] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEADER]: Becoming Leader. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Running, Role: LEADER
I20250905 02:21:47.798760 19256 consensus_queue.cc:237] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } }
I20250905 02:21:47.803226 19143 ts_tablet_manager.cc:1428] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Time spent starting tablet: real 0.067s user 0.031s sys 0.020s
I20250905 02:21:47.813162 19064 ts_manager.cc:194] Registered new tserver with Master: 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1:39371)
I20250905 02:21:47.817281 19064 catalog_manager.cc:5582] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: term changed from 0 to 2, leader changed from <none> to 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1), VOTER 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1) added. New cstate: current_term: 2 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } health_report { overall_health: HEALTHY } } }
W20250905 02:21:47.863236 19064 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b8f259ed87244e51b1e3c174e81cea31 with cas_config_opid_index -1: no extra replica candidate found for tablet b8f259ed87244e51b1e3c174e81cea31 (table TestTable1 [id=c2ff4768efa3495b809568d02a82ccae]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250905 02:21:47.865590 19064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:38777
I20250905 02:21:47.869756 19251 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
W20250905 02:21:48.094988 19255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:48.095517 19255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:48.096071 19255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:48.127362 19255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:48.128264 19255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:21:48.162678 19255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:45857
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=40361
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:48.163947 19255 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:48.165505 19255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:48.177803 19270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:48.180509 19271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:49.433106 19273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:49.435264 19272 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1252 milliseconds
I20250905 02:21:49.435456 19255 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:21:49.436646 19255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:49.439318 19255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:49.440747 19255 hybrid_clock.cc:648] HybridClock initialized: now 1757038909440716 us; error 56 us; skew 500 ppm
I20250905 02:21:49.441519 19255 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:49.447186 19255 webserver.cc:480] Webserver started at http://127.15.105.2:40361/ using document root <none> and password file <none>
I20250905 02:21:49.448113 19255 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:49.448343 19255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:49.456086 19255 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250905 02:21:49.460479 19280 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:49.461524 19255 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250905 02:21:49.461853 19255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ec017dec7c9b42128e05beaa6f879077"
format_stamp: "Formatted at 2025-09-05 02:21:35 on dist-test-slave-ctw4"
I20250905 02:21:49.464444 19255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:49.525434 19255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:49.526818 19255 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:49.527220 19255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:49.529721 19255 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:49.535107 19287 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 02:21:49.542905 19255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 02:21:49.543134 19255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250905 02:21:49.543391 19255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 02:21:49.547856 19255 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 02:21:49.548062 19255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250905 02:21:49.548409 19287 tablet_bootstrap.cc:492] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Bootstrap starting.
I20250905 02:21:49.603251 19287 log.cc:826] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:49.710717 19255 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:45857
I20250905 02:21:49.710784 19394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:45857 every 8 connection(s)
I20250905 02:21:49.714092 19255 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 02:21:49.724007 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 19255
I20250905 02:21:49.725221 19287 tablet_bootstrap.cc:492] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:49.725946 19287 tablet_bootstrap.cc:492] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Bootstrap complete.
I20250905 02:21:49.725869 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:37697
--local_ip_for_outbound_sockets=127.15.105.3
--tserver_master_addrs=127.15.105.62:41029
--webserver_port=46845
--webserver_interface=127.15.105.3
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:21:49.727239 19287 ts_tablet_manager.cc:1397] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Time spent bootstrapping tablet: real 0.179s user 0.141s sys 0.031s
I20250905 02:21:49.742297 19395 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:21:49.742749 19395 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:49.744045 19395 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:21:49.745110 19287 raft_consensus.cc:357] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:21:49.748144 19064 ts_manager.cc:194] Registered new tserver with Master: ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857)
I20250905 02:21:49.748505 19287 raft_consensus.cc:738] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Initialized, Role: FOLLOWER
I20250905 02:21:49.749394 19287 consensus_queue.cc:260] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:21:49.750135 19287 raft_consensus.cc:397] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:49.750494 19287 raft_consensus.cc:491] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:49.750881 19287 raft_consensus.cc:3058] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:21:49.751319 19064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:36149
I20250905 02:21:49.759191 19287 raft_consensus.cc:513] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:21:49.760033 19287 leader_election.cc:304] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec017dec7c9b42128e05beaa6f879077; no voters:
I20250905 02:21:49.761935 19287 leader_election.cc:290] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 02:21:49.762663 19400 raft_consensus.cc:2802] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:21:49.765306 19287 ts_tablet_manager.cc:1428] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Time spent starting tablet: real 0.038s user 0.035s sys 0.004s
I20250905 02:21:49.765450 19395 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
I20250905 02:21:49.766777 19400 raft_consensus.cc:695] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 2 LEADER]: Becoming Leader. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Running, Role: LEADER
I20250905 02:21:49.767488 19400 consensus_queue.cc:237] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:21:49.778604 19064 catalog_manager.cc:5582] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "ec017dec7c9b42128e05beaa6f879077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } health_report { overall_health: HEALTHY } } }
W20250905 02:21:50.039889 19399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:50.040545 19399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:50.041044 19399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:50.071935 19399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:50.073124 19399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:21:50.107251 19399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:37697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=46845
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:50.108570 19399 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:50.110082 19399 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:50.121313 19414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:50.204784 19206 consensus_queue.cc:237] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } }
I20250905 02:21:50.262033 19424 raft_consensus.cc:2953] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } } }
W20250905 02:21:50.290321 19140 consensus_peers.cc:489] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d -> Peer ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857): Couldn't send request to peer ec017dec7c9b42128e05beaa6f879077. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8f259ed87244e51b1e3c174e81cea31. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:21:50.293386 19050 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b8f259ed87244e51b1e3c174e81cea31 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250905 02:21:50.299748 19064 catalog_manager.cc:5582] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: config changed from index -1 to 8, NON_VOTER ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) added. New cstate: current_term: 2 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:21:50.725420 19430 ts_tablet_manager.cc:927] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Initiating tablet copy from peer 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1:39371)
I20250905 02:21:50.733968 19430 tablet_copy_client.cc:323] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: tablet copy: Beginning tablet copy session from remote peer at address 127.15.105.1:39371
I20250905 02:21:50.780020 19226 tablet_copy_service.cc:140] P 6850db245d904b3e977ffc92f91c5c4d: Received BeginTabletCopySession request for tablet b8f259ed87244e51b1e3c174e81cea31 from peer ec017dec7c9b42128e05beaa6f879077 ({username='slave'} at 127.15.105.2:47367)
I20250905 02:21:50.780634 19226 tablet_copy_service.cc:161] P 6850db245d904b3e977ffc92f91c5c4d: Beginning new tablet copy session on tablet b8f259ed87244e51b1e3c174e81cea31 from peer ec017dec7c9b42128e05beaa6f879077 at {username='slave'} at 127.15.105.2:47367: session id = ec017dec7c9b42128e05beaa6f879077-b8f259ed87244e51b1e3c174e81cea31
I20250905 02:21:50.787431 19226 tablet_copy_source_session.cc:215] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:21:50.792119 19430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f259ed87244e51b1e3c174e81cea31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:50.812597 19430 tablet_copy_client.cc:806] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: tablet copy: Starting download of 0 data blocks...
I20250905 02:21:50.813271 19430 tablet_copy_client.cc:670] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: tablet copy: Starting download of 1 WAL segments...
I20250905 02:21:50.822443 19430 tablet_copy_client.cc:538] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:21:50.838227 19430 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Bootstrap starting.
I20250905 02:21:50.997364 19430 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:50.998555 19430 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Bootstrap complete.
I20250905 02:21:50.999302 19430 ts_tablet_manager.cc:1397] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Time spent bootstrapping tablet: real 0.161s user 0.129s sys 0.023s
I20250905 02:21:51.002543 19430 raft_consensus.cc:357] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } }
I20250905 02:21:51.003391 19430 raft_consensus.cc:738] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Initialized, Role: LEARNER
I20250905 02:21:51.004168 19430 consensus_queue.cc:260] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } }
I20250905 02:21:51.013823 19430 ts_tablet_manager.cc:1428] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20250905 02:21:51.028609 19226 tablet_copy_service.cc:342] P 6850db245d904b3e977ffc92f91c5c4d: Request end of tablet copy session ec017dec7c9b42128e05beaa6f879077-b8f259ed87244e51b1e3c174e81cea31 received from {username='slave'} at 127.15.105.2:47367
I20250905 02:21:51.029302 19226 tablet_copy_service.cc:434] P 6850db245d904b3e977ffc92f91c5c4d: ending tablet copy session ec017dec7c9b42128e05beaa6f879077-b8f259ed87244e51b1e3c174e81cea31 on tablet b8f259ed87244e51b1e3c174e81cea31 with peer ec017dec7c9b42128e05beaa6f879077
I20250905 02:21:51.219839 19350 raft_consensus.cc:1215] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20250905 02:21:51.525187 19413 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 19399
W20250905 02:21:51.680826 19413 kernel_stack_watchdog.cc:198] Thread 19399 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:21:51.681653 19399 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.560s user 0.501s sys 1.001s
W20250905 02:21:51.682091 19399 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.560s user 0.501s sys 1.002s
W20250905 02:21:51.682873 19417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:50.121831 19415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:51.683497 19399 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 02:21:51.683598 19416 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1559 milliseconds
I20250905 02:21:51.687284 19399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:21:51.689275 19399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:21:51.690608 19399 hybrid_clock.cc:648] HybridClock initialized: now 1757038911690574 us; error 44 us; skew 500 ppm
I20250905 02:21:51.691357 19399 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:51.697412 19399 webserver.cc:480] Webserver started at http://127.15.105.3:46845/ using document root <none> and password file <none>
I20250905 02:21:51.698299 19399 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:51.698530 19399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:51.706727 19399 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250905 02:21:51.711427 19441 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:51.712527 19399 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250905 02:21:51.712831 19399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2057551e7a3b4d259fc6f55d74fcdbac"
format_stamp: "Formatted at 2025-09-05 02:21:37 on dist-test-slave-ctw4"
I20250905 02:21:51.714746 19399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:51.756548 19442 raft_consensus.cc:1062] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: attempting to promote NON_VOTER ec017dec7c9b42128e05beaa6f879077 to VOTER
I20250905 02:21:51.758210 19442 consensus_queue.cc:237] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:21:51.762655 19350 raft_consensus.cc:1273] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 LEARNER]: Refusing update from remote peer 6850db245d904b3e977ffc92f91c5c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250905 02:21:51.764014 19445 consensus_queue.cc:1035] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250905 02:21:51.766238 19399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:51.767642 19399 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:51.768132 19399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:51.770902 19399 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:51.770404 19442 raft_consensus.cc:2953] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:21:51.772053 19350 raft_consensus.cc:2953] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:21:51.777935 19452 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 02:21:51.784101 19064 catalog_manager.cc:5582] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: config changed from index 8 to 9, ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 02:21:51.787808 19399 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 02:21:51.788130 19399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.002s
I20250905 02:21:51.788456 19399 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 02:21:51.793509 19399 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 02:21:51.793749 19399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.004s
I20250905 02:21:51.794133 19452 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap starting.
I20250905 02:21:51.872097 19452 log.cc:826] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:51.945287 19452 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:51.946195 19452 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap complete.
I20250905 02:21:51.947459 19452 ts_tablet_manager.cc:1397] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent bootstrapping tablet: real 0.154s user 0.112s sys 0.036s
I20250905 02:21:51.962715 19452 raft_consensus.cc:357] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } }
I20250905 02:21:51.964910 19452 raft_consensus.cc:738] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057551e7a3b4d259fc6f55d74fcdbac, State: Initialized, Role: FOLLOWER
I20250905 02:21:51.965852 19452 consensus_queue.cc:260] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } }
I20250905 02:21:51.966531 19452 raft_consensus.cc:397] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:51.966886 19452 raft_consensus.cc:491] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:51.967334 19452 raft_consensus.cc:3058] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:21:51.973620 19452 raft_consensus.cc:513] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } }
I20250905 02:21:51.974468 19452 leader_election.cc:304] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2057551e7a3b4d259fc6f55d74fcdbac; no voters:
I20250905 02:21:51.976361 19399 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:37697
I20250905 02:21:51.976500 19564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:37697 every 8 connection(s)
I20250905 02:21:51.976620 19452 leader_election.cc:290] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 02:21:51.977021 19565 raft_consensus.cc:2802] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:21:51.979820 19399 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 02:21:51.986195 19452 ts_tablet_manager.cc:1428] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent starting tablet: real 0.038s user 0.041s sys 0.000s
I20250905 02:21:51.986248 19565 raft_consensus.cc:695] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEADER]: Becoming Leader. State: Replica: 2057551e7a3b4d259fc6f55d74fcdbac, State: Running, Role: LEADER
I20250905 02:21:51.990120 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 19399
I20250905 02:21:51.990942 19565 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } }
I20250905 02:21:52.009155 19568 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:21:52.009614 19568 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:52.010568 19568 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:21:52.022176 19064 ts_manager.cc:194] Registered new tserver with Master: 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697)
I20250905 02:21:52.023286 19064 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac reported cstate change: term changed from 0 to 2, leader changed from <none> to 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3), VOTER 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) added. New cstate: current_term: 2 leader_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } health_report { overall_health: HEALTHY } } }
I20250905 02:21:52.030742 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:21:52.032920 19064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:54833
I20250905 02:21:52.037209 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:21:52.037649 19568 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
W20250905 02:21:52.042326 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250905 02:21:52.046751 19520 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } }
I20250905 02:21:52.050074 19567 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } }
I20250905 02:21:52.058244 19049 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3292527771b441a49fce17c4d90f3a92 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250905 02:21:52.060334 19446 consensus_peers.cc:489] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac -> Peer 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1:39371): Couldn't send request to peer 6850db245d904b3e977ffc92f91c5c4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3292527771b441a49fce17c4d90f3a92. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:21:52.061086 19063 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac reported cstate change: config changed from index -1 to 8, NON_VOTER 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1) added. New cstate: current_term: 2 leader_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:21:52.069888 19520 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } }
I20250905 02:21:52.072912 19567 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } } }
W20250905 02:21:52.074622 19446 consensus_peers.cc:489] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac -> Peer 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1:39371): Couldn't send request to peer 6850db245d904b3e977ffc92f91c5c4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3292527771b441a49fce17c4d90f3a92. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:21:52.080345 19049 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3292527771b441a49fce17c4d90f3a92 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250905 02:21:52.083220 19447 consensus_peers.cc:489] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac -> Peer ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857): Couldn't send request to peer ec017dec7c9b42128e05beaa6f879077. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3292527771b441a49fce17c4d90f3a92. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:21:52.083349 19063 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac reported cstate change: config changed from index 8 to 9, NON_VOTER ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) added. New cstate: current_term: 2 leader_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:21:52.137975 19206 consensus_queue.cc:237] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: true } }
I20250905 02:21:52.142058 19350 raft_consensus.cc:1273] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Refusing update from remote peer 6850db245d904b3e977ffc92f91c5c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250905 02:21:52.143333 19445 consensus_queue.cc:1035] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250905 02:21:52.148537 19442 raft_consensus.cc:2953] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: true } } }
I20250905 02:21:52.149958 19350 raft_consensus.cc:2953] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: true } } }
W20250905 02:21:52.151679 19138 consensus_peers.cc:489] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d -> Peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Couldn't send request to peer 2057551e7a3b4d259fc6f55d74fcdbac. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8f259ed87244e51b1e3c174e81cea31. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:21:52.155625 19050 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b8f259ed87244e51b1e3c174e81cea31 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250905 02:21:52.158665 19064 catalog_manager.cc:5582] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 reported cstate change: config changed from index 9 to 10, NON_VOTER 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) added. New cstate: current_term: 2 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: true } } }
I20250905 02:21:52.464831 19582 ts_tablet_manager.cc:927] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Initiating tablet copy from peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697)
I20250905 02:21:52.466308 19582 tablet_copy_client.cc:323] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: tablet copy: Beginning tablet copy session from remote peer at address 127.15.105.3:37697
I20250905 02:21:52.475947 19540 tablet_copy_service.cc:140] P 2057551e7a3b4d259fc6f55d74fcdbac: Received BeginTabletCopySession request for tablet 3292527771b441a49fce17c4d90f3a92 from peer ec017dec7c9b42128e05beaa6f879077 ({username='slave'} at 127.15.105.2:58271)
I20250905 02:21:52.476439 19540 tablet_copy_service.cc:161] P 2057551e7a3b4d259fc6f55d74fcdbac: Beginning new tablet copy session on tablet 3292527771b441a49fce17c4d90f3a92 from peer ec017dec7c9b42128e05beaa6f879077 at {username='slave'} at 127.15.105.2:58271: session id = ec017dec7c9b42128e05beaa6f879077-3292527771b441a49fce17c4d90f3a92
I20250905 02:21:52.481163 19540 tablet_copy_source_session.cc:215] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:21:52.484339 19582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3292527771b441a49fce17c4d90f3a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:52.494223 19582 tablet_copy_client.cc:806] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: tablet copy: Starting download of 0 data blocks...
I20250905 02:21:52.494699 19582 tablet_copy_client.cc:670] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: tablet copy: Starting download of 1 WAL segments...
I20250905 02:21:52.498382 19582 tablet_copy_client.cc:538] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:21:52.503724 19582 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Bootstrap starting.
I20250905 02:21:52.563342 19050 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b8f259ed87244e51b1e3c174e81cea31 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250905 02:21:52.589243 19582 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:52.589843 19582 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Bootstrap complete.
I20250905 02:21:52.590255 19582 ts_tablet_manager.cc:1397] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Time spent bootstrapping tablet: real 0.087s user 0.075s sys 0.012s
I20250905 02:21:52.591636 19582 raft_consensus.cc:357] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } }
I20250905 02:21:52.592118 19582 raft_consensus.cc:738] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Initialized, Role: LEARNER
I20250905 02:21:52.592486 19582 consensus_queue.cc:260] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } }
I20250905 02:21:52.594018 19582 ts_tablet_manager.cc:1428] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 02:21:52.595515 19540 tablet_copy_service.cc:342] P 2057551e7a3b4d259fc6f55d74fcdbac: Request end of tablet copy session ec017dec7c9b42128e05beaa6f879077-3292527771b441a49fce17c4d90f3a92 received from {username='slave'} at 127.15.105.2:58271
I20250905 02:21:52.595902 19540 tablet_copy_service.cc:434] P 2057551e7a3b4d259fc6f55d74fcdbac: ending tablet copy session ec017dec7c9b42128e05beaa6f879077-3292527771b441a49fce17c4d90f3a92 on tablet 3292527771b441a49fce17c4d90f3a92 with peer ec017dec7c9b42128e05beaa6f879077
I20250905 02:21:52.601649 19586 ts_tablet_manager.cc:927] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Initiating tablet copy from peer 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1:39371)
I20250905 02:21:52.603677 19586 tablet_copy_client.cc:323] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: tablet copy: Beginning tablet copy session from remote peer at address 127.15.105.1:39371
I20250905 02:21:52.605041 19226 tablet_copy_service.cc:140] P 6850db245d904b3e977ffc92f91c5c4d: Received BeginTabletCopySession request for tablet b8f259ed87244e51b1e3c174e81cea31 from peer 2057551e7a3b4d259fc6f55d74fcdbac ({username='slave'} at 127.15.105.3:41347)
I20250905 02:21:52.605393 19226 tablet_copy_service.cc:161] P 6850db245d904b3e977ffc92f91c5c4d: Beginning new tablet copy session on tablet b8f259ed87244e51b1e3c174e81cea31 from peer 2057551e7a3b4d259fc6f55d74fcdbac at {username='slave'} at 127.15.105.3:41347: session id = 2057551e7a3b4d259fc6f55d74fcdbac-b8f259ed87244e51b1e3c174e81cea31
I20250905 02:21:52.609444 19226 tablet_copy_source_session.cc:215] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:21:52.611848 19586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8f259ed87244e51b1e3c174e81cea31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:52.623404 19586 tablet_copy_client.cc:806] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: tablet copy: Starting download of 0 data blocks...
I20250905 02:21:52.623832 19586 tablet_copy_client.cc:670] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: tablet copy: Starting download of 1 WAL segments...
I20250905 02:21:52.626914 19586 tablet_copy_client.cc:538] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:21:52.629892 19588 ts_tablet_manager.cc:927] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Initiating tablet copy from peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697)
I20250905 02:21:52.631927 19588 tablet_copy_client.cc:323] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.105.3:37697
I20250905 02:21:52.632655 19586 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap starting.
I20250905 02:21:52.633293 19540 tablet_copy_service.cc:140] P 2057551e7a3b4d259fc6f55d74fcdbac: Received BeginTabletCopySession request for tablet 3292527771b441a49fce17c4d90f3a92 from peer 6850db245d904b3e977ffc92f91c5c4d ({username='slave'} at 127.15.105.1:39801)
I20250905 02:21:52.633780 19540 tablet_copy_service.cc:161] P 2057551e7a3b4d259fc6f55d74fcdbac: Beginning new tablet copy session on tablet 3292527771b441a49fce17c4d90f3a92 from peer 6850db245d904b3e977ffc92f91c5c4d at {username='slave'} at 127.15.105.1:39801: session id = 6850db245d904b3e977ffc92f91c5c4d-3292527771b441a49fce17c4d90f3a92
I20250905 02:21:52.640517 19540 tablet_copy_source_session.cc:215] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:21:52.643278 19588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3292527771b441a49fce17c4d90f3a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:52.653272 19588 tablet_copy_client.cc:806] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: tablet copy: Starting download of 0 data blocks...
I20250905 02:21:52.653795 19588 tablet_copy_client.cc:670] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: tablet copy: Starting download of 1 WAL segments...
I20250905 02:21:52.657215 19588 tablet_copy_client.cc:538] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:21:52.662328 19588 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap starting.
I20250905 02:21:52.710259 19586 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:52.710850 19586 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap complete.
I20250905 02:21:52.711303 19586 ts_tablet_manager.cc:1397] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent bootstrapping tablet: real 0.079s user 0.071s sys 0.004s
I20250905 02:21:52.713003 19586 raft_consensus.cc:357] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: true } }
I20250905 02:21:52.713460 19586 raft_consensus.cc:738] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2057551e7a3b4d259fc6f55d74fcdbac, State: Initialized, Role: LEARNER
I20250905 02:21:52.713882 19586 consensus_queue.cc:260] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: NON_VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: true } }
I20250905 02:21:52.715937 19586 ts_tablet_manager.cc:1428] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent starting tablet: real 0.004s user 0.008s sys 0.000s
I20250905 02:21:52.717447 19226 tablet_copy_service.cc:342] P 6850db245d904b3e977ffc92f91c5c4d: Request end of tablet copy session 2057551e7a3b4d259fc6f55d74fcdbac-b8f259ed87244e51b1e3c174e81cea31 received from {username='slave'} at 127.15.105.3:41347
I20250905 02:21:52.717834 19226 tablet_copy_service.cc:434] P 6850db245d904b3e977ffc92f91c5c4d: ending tablet copy session 2057551e7a3b4d259fc6f55d74fcdbac-b8f259ed87244e51b1e3c174e81cea31 on tablet b8f259ed87244e51b1e3c174e81cea31 with peer 2057551e7a3b4d259fc6f55d74fcdbac
I20250905 02:21:52.745435 19588 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:52.746227 19588 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap complete.
I20250905 02:21:52.746788 19588 ts_tablet_manager.cc:1397] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Time spent bootstrapping tablet: real 0.085s user 0.071s sys 0.012s
I20250905 02:21:52.748956 19588 raft_consensus.cc:357] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } }
I20250905 02:21:52.749576 19588 raft_consensus.cc:738] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Initialized, Role: LEARNER
I20250905 02:21:52.751324 19588 consensus_queue.cc:260] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: NON_VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: true } }
I20250905 02:21:52.753219 19588 ts_tablet_manager.cc:1428] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250905 02:21:52.755239 19540 tablet_copy_service.cc:342] P 2057551e7a3b4d259fc6f55d74fcdbac: Request end of tablet copy session 6850db245d904b3e977ffc92f91c5c4d-3292527771b441a49fce17c4d90f3a92 received from {username='slave'} at 127.15.105.1:39801
I20250905 02:21:52.755599 19540 tablet_copy_service.cc:434] P 2057551e7a3b4d259fc6f55d74fcdbac: ending tablet copy session 6850db245d904b3e977ffc92f91c5c4d-3292527771b441a49fce17c4d90f3a92 on tablet 3292527771b441a49fce17c4d90f3a92 with peer 6850db245d904b3e977ffc92f91c5c4d
I20250905 02:21:52.932159 19350 raft_consensus.cc:1215] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250905 02:21:53.022691 19520 raft_consensus.cc:1215] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250905 02:21:53.046904 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6850db245d904b3e977ffc92f91c5c4d to finish bootstrapping
I20250905 02:21:53.064433 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ec017dec7c9b42128e05beaa6f879077 to finish bootstrapping
I20250905 02:21:53.076004 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2057551e7a3b4d259fc6f55d74fcdbac to finish bootstrapping
I20250905 02:21:53.229944 19206 raft_consensus.cc:1215] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250905 02:21:53.330261 19185 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 02:21:53.331285 19499 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 02:21:53.337267 19330 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 02:21:53.499939 19565 raft_consensus.cc:1062] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: attempting to promote NON_VOTER ec017dec7c9b42128e05beaa6f879077 to VOTER
I20250905 02:21:53.501472 19565 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:21:53.515991 19350 raft_consensus.cc:1273] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 LEARNER]: Refusing update from remote peer 2057551e7a3b4d259fc6f55d74fcdbac: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250905 02:21:53.517504 19593 consensus_queue.cc:1035] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250905 02:21:53.521250 19206 raft_consensus.cc:1273] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEARNER]: Refusing update from remote peer 2057551e7a3b4d259fc6f55d74fcdbac: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250905 02:21:53.523386 19593 consensus_queue.cc:1035] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250905 02:21:53.537436 19593 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:21:53.540575 19205 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:21:53.546747 19349 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:21:53.551710 19062 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: config changed from index 9 to 10, ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:21:53.559240 19593 raft_consensus.cc:1062] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: attempting to promote NON_VOTER 6850db245d904b3e977ffc92f91c5c4d to VOTER
I20250905 02:21:53.563860 19565 raft_consensus.cc:1062] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: attempting to promote NON_VOTER 6850db245d904b3e977ffc92f91c5c4d to VOTER
I20250905 02:21:53.565716 19593 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:21:53.572692 19205 raft_consensus.cc:1273] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEARNER]: Refusing update from remote peer 2057551e7a3b4d259fc6f55d74fcdbac: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250905 02:21:53.576354 19592 consensus_queue.cc:1035] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20250905 02:21:53.581107 19349 raft_consensus.cc:1273] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Refusing update from remote peer 2057551e7a3b4d259fc6f55d74fcdbac: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5b4815f1687a4367b7328097a0d48ada | 127.15.105.62:41029 | HEALTHY
Unusual flags for Master:
I20250905 02:21:53.587559 19592 consensus_queue.cc:1035] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.105.20:33379 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
2057551e7a3b4d259fc6f55d74fcdbac | 127.15.105.3:37697 | HEALTHY | <none> | 1 | 0
6850db245d904b3e977ffc92f91c5c4d | 127.15.105.1:39371 | HEALTHY | <none> | 1 | 0
ec017dec7c9b42128e05beaa6f879077 | 127.15.105.2:45857 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
W20250905 02:21:53.598312 19565 raft_consensus.cc:1066] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Unable to promote non-voter 6850db245d904b3e977ffc92f91c5c4d: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: NON_VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: true } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }.
Pending config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:21:53.599005 19565 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:21:53.600697 19349 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
I20250905 02:21:53.613057 19595 raft_consensus.cc:1062] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: attempting to promote NON_VOTER 2057551e7a3b4d259fc6f55d74fcdbac to VOTER
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.105.1 | experimental | 127.15.105.1:39371
local_ip_for_outbound_sockets | 127.15.105.2 | experimental | 127.15.105.2:45857
local_ip_for_outbound_sockets | 127.15.105.3 | experimental | 127.15.105.3:37697
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.105.1:39371
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.105.2:45857
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.105.3:37697
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.105.20:33379 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
I20250905 02:21:53.615630 19595 consensus_queue.cc:237] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
I20250905 02:21:53.616581 19205 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 02:21:53.633191 19520 raft_consensus.cc:1273] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 LEARNER]: Refusing update from remote peer 6850db245d904b3e977ffc92f91c5c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250905 02:21:53.633575 15780 log_verifier.cc:126] Checking tablet 3292527771b441a49fce17c4d90f3a92
I20250905 02:21:53.637214 19634 consensus_queue.cc:1035] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250905 02:21:53.637217 19063 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: config changed from index 10 to 11, 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:21:53.643106 19349 raft_consensus.cc:1273] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Refusing update from remote peer 6850db245d904b3e977ffc92f91c5c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250905 02:21:53.644435 19634 consensus_queue.cc:1035] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250905 02:21:53.651695 19634 raft_consensus.cc:2953] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } } }
I20250905 02:21:53.653445 19350 raft_consensus.cc:2953] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } } }
I20250905 02:21:53.655534 19519 raft_consensus.cc:2953] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } } }
I20250905 02:21:53.668044 19064 catalog_manager.cc:5582] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 reported cstate change: config changed from index 10 to 11, 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } } }
I20250905 02:21:53.764700 15780 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 3292527771b441a49fce17c4d90f3a92
I20250905 02:21:53.765084 15780 log_verifier.cc:126] Checking tablet b433bc2d1fc847dbb0e072e941f0f0cb
I20250905 02:21:53.795940 15780 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b433bc2d1fc847dbb0e072e941f0f0cb
I20250905 02:21:53.796165 15780 log_verifier.cc:126] Checking tablet b8f259ed87244e51b1e3c174e81cea31
I20250905 02:21:53.874402 15780 log_verifier.cc:177] Verified matching terms for 11 ops in tablet b8f259ed87244e51b1e3c174e81cea31
I20250905 02:21:53.874835 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 19031
I20250905 02:21:53.903690 15780 minidump.cc:252] Setting minidump size limit to 20M
I20250905 02:21:53.905030 15780 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:53.906024 15780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:53.915930 19641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:53.917068 19644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:53.917768 15780 server_base.cc:1047] running on GCE node
W20250905 02:21:53.916875 19642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:54.015522 15780 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:54.016090 15780 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:54.016363 15780 hybrid_clock.cc:648] HybridClock initialized: now 1757038914016341 us; error 0 us; skew 500 ppm
I20250905 02:21:54.017211 15780 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:54.021032 15780 webserver.cc:480] Webserver started at http://0.0.0.0:37069/ using document root <none> and password file <none>
I20250905 02:21:54.021777 15780 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:54.021948 15780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:54.026880 15780 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 02:21:54.030135 19649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:54.030968 15780 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:21:54.031262 15780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5b4815f1687a4367b7328097a0d48ada"
format_stamp: "Formatted at 2025-09-05 02:21:31 on dist-test-slave-ctw4"
I20250905 02:21:54.033378 15780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:54.047238 15780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:54.048578 15780 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:21:54.048970 15780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:54.056771 15780 sys_catalog.cc:263] Verifying existing consensus state
W20250905 02:21:54.060021 15780 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.15.105.62:41029 exists but no master address supplied!
I20250905 02:21:54.061678 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap starting.
I20250905 02:21:54.102800 15780 log.cc:826] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Log is configured to *not* fsync() on all Append() calls
I20250905 02:21:54.164393 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:21:54.165130 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap complete.
I20250905 02:21:54.177889 15780 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:54.178453 15780 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4815f1687a4367b7328097a0d48ada, State: Initialized, Role: FOLLOWER
I20250905 02:21:54.179122 15780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:54.179575 15780 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:54.179857 15780 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:54.180156 15780 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 3 FOLLOWER]: Advancing to term 4
I20250905 02:21:54.185359 15780 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:54.186033 15780 leader_election.cc:304] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4815f1687a4367b7328097a0d48ada; no voters:
I20250905 02:21:54.187153 15780 leader_election.cc:290] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [CANDIDATE]: Term 4 election: Requested vote from peers
I20250905 02:21:54.187407 19656 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 4 FOLLOWER]: Leader election won for term 4
I20250905 02:21:54.188795 19656 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 4 LEADER]: Becoming Leader. State: Replica: 5b4815f1687a4367b7328097a0d48ada, State: Running, Role: LEADER
I20250905 02:21:54.189564 19656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:21:54.196894 19657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "5b4815f1687a4367b7328097a0d48ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } } }
I20250905 02:21:54.196710 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4815f1687a4367b7328097a0d48ada. Latest consensus state: current_term: 4 leader_uuid: "5b4815f1687a4367b7328097a0d48ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } } }
I20250905 02:21:54.197398 19657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:54.197441 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:54.230113 15780 tablet_replica.cc:331] stopping tablet replica
I20250905 02:21:54.230777 15780 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 4 LEADER]: Raft consensus shutting down.
I20250905 02:21:54.231304 15780 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:54.234031 15780 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250905 02:21:54.234586 15780 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250905 02:21:54.259243 15780 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250905 02:21:54.690305 19395 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:41029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:41029: connect: Connection refused (error 111)
W20250905 02:21:54.726738 19568 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:41029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:41029: connect: Connection refused (error 111)
W20250905 02:21:54.735211 19251 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:41029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:41029: connect: Connection refused (error 111)
I20250905 02:21:59.539837 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 19101
I20250905 02:21:59.565613 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 19255
I20250905 02:21:59.592450 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 19399
I20250905 02:21:59.621517 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:41029
--webserver_interface=127.15.105.62
--webserver_port=41509
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:41029 with env {}
W20250905 02:21:59.918427 19729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:21:59.919009 19729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:21:59.919474 19729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:21:59.949584 19729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:21:59.949872 19729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:21:59.950158 19729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:21:59.950395 19729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:21:59.984257 19729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:41029
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=41509
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:21:59.985530 19729 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:21:59.987092 19729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:59.997298 19735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:59.997941 19736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:01.190007 19737 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1189 milliseconds
W20250905 02:22:01.190865 19738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:01.194038 19729 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.196s user 0.378s sys 0.818s
W20250905 02:22:01.194308 19729 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.197s user 0.378s sys 0.818s
I20250905 02:22:01.194514 19729 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:01.195565 19729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:01.198089 19729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:01.199435 19729 hybrid_clock.cc:648] HybridClock initialized: now 1757038921199395 us; error 49 us; skew 500 ppm
I20250905 02:22:01.200248 19729 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:01.207938 19729 webserver.cc:480] Webserver started at http://127.15.105.62:41509/ using document root <none> and password file <none>
I20250905 02:22:01.208916 19729 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:01.209136 19729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:01.217346 19729 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20250905 02:22:01.222674 19745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:01.223765 19729 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.005s
I20250905 02:22:01.224131 19729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5b4815f1687a4367b7328097a0d48ada"
format_stamp: "Formatted at 2025-09-05 02:21:31 on dist-test-slave-ctw4"
I20250905 02:22:01.226022 19729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:01.302596 19729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:01.304072 19729 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:01.304472 19729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:01.373191 19729 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:41029
I20250905 02:22:01.373260 19796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:41029 every 8 connection(s)
I20250905 02:22:01.376368 19729 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:22:01.385416 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 19729
I20250905 02:22:01.386818 19797 sys_catalog.cc:263] Verifying existing consensus state
I20250905 02:22:01.387225 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:39371
--local_ip_for_outbound_sockets=127.15.105.1
--tserver_master_addrs=127.15.105.62:41029
--webserver_port=32773
--webserver_interface=127.15.105.1
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:22:01.392074 19797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap starting.
I20250905 02:22:01.404790 19797 log.cc:826] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:01.502975 19797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:01.503737 19797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Bootstrap complete.
I20250905 02:22:01.522245 19797 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:22:01.524289 19797 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4815f1687a4367b7328097a0d48ada, State: Initialized, Role: FOLLOWER
I20250905 02:22:01.525063 19797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:22:01.525520 19797 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:01.525766 19797 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:01.526049 19797 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 5 FOLLOWER]: Advancing to term 6
I20250905 02:22:01.531298 19797 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:22:01.532009 19797 leader_election.cc:304] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4815f1687a4367b7328097a0d48ada; no voters:
I20250905 02:22:01.534085 19797 leader_election.cc:290] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [CANDIDATE]: Term 6 election: Requested vote from peers
I20250905 02:22:01.534369 19801 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 6 FOLLOWER]: Leader election won for term 6
I20250905 02:22:01.537631 19801 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [term 6 LEADER]: Becoming Leader. State: Replica: 5b4815f1687a4367b7328097a0d48ada, State: Running, Role: LEADER
I20250905 02:22:01.538471 19801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } }
I20250905 02:22:01.539106 19797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:22:01.546873 19803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4815f1687a4367b7328097a0d48ada. Latest consensus state: current_term: 6 leader_uuid: "5b4815f1687a4367b7328097a0d48ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } } }
I20250905 02:22:01.548130 19802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "5b4815f1687a4367b7328097a0d48ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4815f1687a4367b7328097a0d48ada" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 41029 } } }
I20250905 02:22:01.548780 19802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:01.550495 19803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:01.560753 19808 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:22:01.577171 19808 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=194eb01f9c6249879c384f0c773ebc97]
I20250905 02:22:01.579440 19808 catalog_manager.cc:671] Loaded metadata for table TestTable [id=b569ba47a7fd47fcb1f4d475b0959129]
I20250905 02:22:01.581553 19808 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=c2ff4768efa3495b809568d02a82ccae]
I20250905 02:22:01.595508 19808 tablet_loader.cc:96] loaded metadata for tablet 3292527771b441a49fce17c4d90f3a92 (table TestTable [id=b569ba47a7fd47fcb1f4d475b0959129])
I20250905 02:22:01.597028 19808 tablet_loader.cc:96] loaded metadata for tablet b433bc2d1fc847dbb0e072e941f0f0cb (table TestTable2 [id=194eb01f9c6249879c384f0c773ebc97])
I20250905 02:22:01.598568 19808 tablet_loader.cc:96] loaded metadata for tablet b8f259ed87244e51b1e3c174e81cea31 (table TestTable1 [id=c2ff4768efa3495b809568d02a82ccae])
I20250905 02:22:01.600083 19808 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:22:01.605836 19808 catalog_manager.cc:1261] Loaded cluster ID: dc398bad0ff543d888d4b46d29e02845
I20250905 02:22:01.606155 19808 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:22:01.615034 19808 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:22:01.620913 19808 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5b4815f1687a4367b7328097a0d48ada: Loaded TSK: 0
I20250905 02:22:01.622650 19808 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250905 02:22:01.758285 19799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:01.758797 19799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:01.759282 19799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:01.790541 19799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:01.791435 19799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:22:01.826598 19799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:39371
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=32773
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:01.827975 19799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:01.829504 19799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:01.841852 19824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:03.246210 19823 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 19799
W20250905 02:22:03.595065 19823 kernel_stack_watchdog.cc:198] Thread 19799 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:22:03.600018 19799 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.757s user 0.481s sys 1.044s
W20250905 02:22:03.600459 19799 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.758s user 0.481s sys 1.045s
W20250905 02:22:03.602085 19826 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1759 milliseconds
W20250905 02:22:03.602362 19827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:01.843185 19825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:03.602679 19799 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:03.607582 19799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:03.610116 19799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:03.611634 19799 hybrid_clock.cc:648] HybridClock initialized: now 1757038923611588 us; error 43 us; skew 500 ppm
I20250905 02:22:03.612711 19799 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:03.620133 19799 webserver.cc:480] Webserver started at http://127.15.105.1:32773/ using document root <none> and password file <none>
I20250905 02:22:03.621383 19799 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:03.621660 19799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:03.632118 19799 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.002s
I20250905 02:22:03.636943 19835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:03.638020 19799 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250905 02:22:03.638336 19799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6850db245d904b3e977ffc92f91c5c4d"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-ctw4"
I20250905 02:22:03.640292 19799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:03.701464 19799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:03.703207 19799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:03.703648 19799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:03.706277 19799 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:03.712426 19842 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250905 02:22:03.731082 19799 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250905 02:22:03.731343 19799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.001s sys 0.002s
I20250905 02:22:03.731611 19799 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250905 02:22:03.736857 19842 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap starting.
I20250905 02:22:03.739401 19799 ts_tablet_manager.cc:610] Registered 2 tablets
I20250905 02:22:03.739607 19799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.005s sys 0.000s
I20250905 02:22:03.812557 19842 log.cc:826] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:03.925974 19799 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:39371
I20250905 02:22:03.926178 19949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:39371 every 8 connection(s)
I20250905 02:22:03.929457 19799 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:22:03.931304 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 19799
I20250905 02:22:03.933552 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:45857
--local_ip_for_outbound_sockets=127.15.105.2
--tserver_master_addrs=127.15.105.62:41029
--webserver_port=40361
--webserver_interface=127.15.105.2
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:22:03.948048 19842 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:03.949478 19842 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap complete.
I20250905 02:22:03.951771 19842 ts_tablet_manager.cc:1397] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Time spent bootstrapping tablet: real 0.215s user 0.164s sys 0.044s
I20250905 02:22:03.979679 19842 raft_consensus.cc:357] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:03.982962 19842 raft_consensus.cc:738] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Initialized, Role: FOLLOWER
I20250905 02:22:03.984088 19842 consensus_queue.cc:260] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:03.999686 19950 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:22:03.999914 19842 ts_tablet_manager.cc:1428] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d: Time spent starting tablet: real 0.048s user 0.026s sys 0.011s
I20250905 02:22:04.000213 19950 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:04.000605 19842 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap starting.
I20250905 02:22:04.001417 19950 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:22:04.006475 19762 ts_manager.cc:194] Registered new tserver with Master: 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1:39371)
I20250905 02:22:04.011117 19762 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) added, VOTER 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1) added, VOTER ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:22:04.098342 19762 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:42273
I20250905 02:22:04.103435 19950 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
I20250905 02:22:04.137773 19842 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:04.138666 19842 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Bootstrap complete.
I20250905 02:22:04.140221 19842 ts_tablet_manager.cc:1397] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Time spent bootstrapping tablet: real 0.140s user 0.110s sys 0.027s
I20250905 02:22:04.142345 19842 raft_consensus.cc:357] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:04.142956 19842 raft_consensus.cc:738] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Initialized, Role: FOLLOWER
I20250905 02:22:04.143536 19842 consensus_queue.cc:260] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:04.145341 19842 ts_tablet_manager.cc:1428] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250905 02:22:04.311632 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:04.312183 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:04.312654 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:04.345491 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:04.346366 19951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:22:04.391340 19951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:45857
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=40361
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:04.392961 19951 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:04.394984 19951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:04.409253 19966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:05.467271 19972 raft_consensus.cc:491] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:05.468276 19972 raft_consensus.cc:513] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
W20250905 02:22:05.478647 19837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111)
I20250905 02:22:05.483256 19972 leader_election.cc:290] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697), ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857)
W20250905 02:22:05.491674 19839 leader_election.cc:336] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857): Network error: Client connection negotiation failed: client connection to 127.15.105.2:45857: connect: Connection refused (error 111)
W20250905 02:22:05.493249 19837 leader_election.cc:336] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111)
I20250905 02:22:05.493894 19837 leader_election.cc:304] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6850db245d904b3e977ffc92f91c5c4d; no voters: 2057551e7a3b4d259fc6f55d74fcdbac, ec017dec7c9b42128e05beaa6f879077
I20250905 02:22:05.495272 19972 raft_consensus.cc:2747] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 02:22:05.579892 19972 raft_consensus.cc:491] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:05.580542 19972 raft_consensus.cc:513] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
W20250905 02:22:05.595187 19839 leader_election.cc:336] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857): Network error: Client connection negotiation failed: client connection to 127.15.105.2:45857: connect: Connection refused (error 111)
I20250905 02:22:05.596139 19972 leader_election.cc:290] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857), 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697)
W20250905 02:22:05.605405 19837 leader_election.cc:336] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111)
I20250905 02:22:05.606019 19837 leader_election.cc:304] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6850db245d904b3e977ffc92f91c5c4d; no voters: 2057551e7a3b4d259fc6f55d74fcdbac, ec017dec7c9b42128e05beaa6f879077
I20250905 02:22:05.606813 19972 raft_consensus.cc:2747] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 02:22:04.412349 19967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:06.239301 19951 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.827s user 0.704s sys 1.119s
W20250905 02:22:06.239734 19951 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.828s user 0.704s sys 1.119s
W20250905 02:22:05.812124 19965 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 19951
W20250905 02:22:06.241874 19969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:06.244118 19968 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1827 milliseconds
I20250905 02:22:06.244179 19951 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:06.245260 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:06.247152 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:06.248499 19951 hybrid_clock.cc:648] HybridClock initialized: now 1757038926248456 us; error 43 us; skew 500 ppm
I20250905 02:22:06.249239 19951 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:06.254837 19951 webserver.cc:480] Webserver started at http://127.15.105.2:40361/ using document root <none> and password file <none>
I20250905 02:22:06.255709 19951 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:06.255961 19951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:06.263341 19951 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250905 02:22:06.267899 19980 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:06.268810 19951 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250905 02:22:06.269129 19951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ec017dec7c9b42128e05beaa6f879077"
format_stamp: "Formatted at 2025-09-05 02:21:35 on dist-test-slave-ctw4"
I20250905 02:22:06.270984 19951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:06.338013 19951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:06.339469 19951 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:06.339916 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:06.342489 19951 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:06.348223 19987 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250905 02:22:06.363940 19951 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250905 02:22:06.364205 19951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.002s
I20250905 02:22:06.364475 19951 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250905 02:22:06.369354 19987 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Bootstrap starting.
I20250905 02:22:06.375525 19951 ts_tablet_manager.cc:610] Registered 3 tablets
I20250905 02:22:06.375738 19951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.007s sys 0.001s
I20250905 02:22:06.430382 19987 log.cc:826] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:06.537985 19951 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:45857
I20250905 02:22:06.538103 20094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:45857 every 8 connection(s)
I20250905 02:22:06.541394 19951 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 02:22:06.546828 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 19951
I20250905 02:22:06.549329 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:37697
--local_ip_for_outbound_sockets=127.15.105.3
--tserver_master_addrs=127.15.105.62:41029
--webserver_port=46845
--webserver_interface=127.15.105.3
--builtin_ntp_servers=127.15.105.20:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 02:22:06.561978 19987 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:06.562942 19987 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Bootstrap complete.
I20250905 02:22:06.564798 19987 ts_tablet_manager.cc:1397] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Time spent bootstrapping tablet: real 0.196s user 0.161s sys 0.032s
I20250905 02:22:06.584036 20095 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:22:06.584560 20095 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:06.585783 20095 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:22:06.583559 19987 raft_consensus.cc:357] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:06.586616 19987 raft_consensus.cc:738] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Initialized, Role: FOLLOWER
I20250905 02:22:06.587370 19987 consensus_queue.cc:260] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:06.591233 19987 ts_tablet_manager.cc:1428] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Time spent starting tablet: real 0.026s user 0.021s sys 0.000s
I20250905 02:22:06.591861 19987 tablet_bootstrap.cc:492] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Bootstrap starting.
I20250905 02:22:06.596580 19761 ts_manager.cc:194] Registered new tserver with Master: ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857)
I20250905 02:22:06.601928 19761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:55267
I20250905 02:22:06.605999 20095 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
I20250905 02:22:06.675622 19987 tablet_bootstrap.cc:492] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:06.676301 19987 tablet_bootstrap.cc:492] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Bootstrap complete.
I20250905 02:22:06.677449 19987 ts_tablet_manager.cc:1397] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Time spent bootstrapping tablet: real 0.086s user 0.060s sys 0.023s
I20250905 02:22:06.678958 19987 raft_consensus.cc:357] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:22:06.679306 19987 raft_consensus.cc:738] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Initialized, Role: FOLLOWER
I20250905 02:22:06.679760 19987 consensus_queue.cc:260] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:22:06.680187 19987 raft_consensus.cc:397] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:06.680444 19987 raft_consensus.cc:491] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:06.680712 19987 raft_consensus.cc:3058] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:06.685633 19987 raft_consensus.cc:513] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:22:06.686239 19987 leader_election.cc:304] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec017dec7c9b42128e05beaa6f879077; no voters:
I20250905 02:22:06.686712 19987 leader_election.cc:290] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250905 02:22:06.686932 20100 raft_consensus.cc:2802] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:22:06.690701 19987 ts_tablet_manager.cc:1428] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20250905 02:22:06.690997 20100 raft_consensus.cc:695] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [term 3 LEADER]: Becoming Leader. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Running, Role: LEADER
I20250905 02:22:06.691516 19987 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Bootstrap starting.
I20250905 02:22:06.691817 20100 consensus_queue.cc:237] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } }
I20250905 02:22:06.700591 19761 catalog_manager.cc:5582] T b433bc2d1fc847dbb0e072e941f0f0cb P ec017dec7c9b42128e05beaa6f879077 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "ec017dec7c9b42128e05beaa6f879077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:06.788087 19987 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:06.788759 19987 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Bootstrap complete.
I20250905 02:22:06.789906 19987 ts_tablet_manager.cc:1397] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Time spent bootstrapping tablet: real 0.099s user 0.084s sys 0.012s
I20250905 02:22:06.791401 19987 raft_consensus.cc:357] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:06.791877 19987 raft_consensus.cc:738] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec017dec7c9b42128e05beaa6f879077, State: Initialized, Role: FOLLOWER
I20250905 02:22:06.792393 19987 consensus_queue.cc:260] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:06.793900 19987 ts_tablet_manager.cc:1428] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
W20250905 02:22:06.891490 20099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:06.892057 20099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:06.892554 20099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:06.923951 20099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:06.924798 20099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:22:06.960496 20099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:37697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=46845
--tserver_master_addrs=127.15.105.62:41029
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:06.961840 20099 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:06.963433 20099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:06.974789 20116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:07.514143 20122 raft_consensus.cc:491] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:07.514850 20122 raft_consensus.cc:513] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:07.526496 20122 leader_election.cc:290] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697), ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857)
W20250905 02:22:07.553908 19837 leader_election.cc:336] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111)
I20250905 02:22:07.556779 20050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3292527771b441a49fce17c4d90f3a92" candidate_uuid: "6850db245d904b3e977ffc92f91c5c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ec017dec7c9b42128e05beaa6f879077" is_pre_election: true
I20250905 02:22:07.557662 20050 raft_consensus.cc:2466] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6850db245d904b3e977ffc92f91c5c4d in term 2.
I20250905 02:22:07.559631 19839 leader_election.cc:304] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6850db245d904b3e977ffc92f91c5c4d, ec017dec7c9b42128e05beaa6f879077; no voters: 2057551e7a3b4d259fc6f55d74fcdbac
I20250905 02:22:07.561079 20122 raft_consensus.cc:2802] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 02:22:07.561524 20122 raft_consensus.cc:491] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:07.561911 20122 raft_consensus.cc:3058] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:07.582180 20122 raft_consensus.cc:513] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:07.589356 20050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3292527771b441a49fce17c4d90f3a92" candidate_uuid: "6850db245d904b3e977ffc92f91c5c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ec017dec7c9b42128e05beaa6f879077"
I20250905 02:22:07.590148 20050 raft_consensus.cc:3058] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Advancing to term 3
W20250905 02:22:07.597577 19837 leader_election.cc:336] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111)
I20250905 02:22:07.602207 20050 raft_consensus.cc:2466] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6850db245d904b3e977ffc92f91c5c4d in term 3.
I20250905 02:22:07.602824 20122 leader_election.cc:290] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 election: Requested vote from peers 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697), ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857)
I20250905 02:22:07.616961 19839 leader_election.cc:304] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6850db245d904b3e977ffc92f91c5c4d, ec017dec7c9b42128e05beaa6f879077; no voters: 2057551e7a3b4d259fc6f55d74fcdbac
I20250905 02:22:07.618472 20122 raft_consensus.cc:2802] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:22:07.622711 20122 raft_consensus.cc:695] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 3 LEADER]: Becoming Leader. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Running, Role: LEADER
I20250905 02:22:07.624004 20122 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:07.650017 19761 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: term changed from 2 to 3, leader changed from <none> to 6850db245d904b3e977ffc92f91c5c4d (127.15.105.1). New cstate: current_term: 3 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:07.817962 20122 raft_consensus.cc:491] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:07.818679 20122 raft_consensus.cc:513] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:07.827764 20050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8f259ed87244e51b1e3c174e81cea31" candidate_uuid: "6850db245d904b3e977ffc92f91c5c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ec017dec7c9b42128e05beaa6f879077" is_pre_election: true
I20250905 02:22:07.828680 20050 raft_consensus.cc:2466] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6850db245d904b3e977ffc92f91c5c4d in term 2.
I20250905 02:22:07.831399 19839 leader_election.cc:304] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6850db245d904b3e977ffc92f91c5c4d, ec017dec7c9b42128e05beaa6f879077; no voters:
I20250905 02:22:07.832806 20127 raft_consensus.cc:2802] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 02:22:07.833321 20127 raft_consensus.cc:491] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:07.833748 20127 raft_consensus.cc:3058] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 2 FOLLOWER]: Advancing to term 3
W20250905 02:22:07.844390 19837 leader_election.cc:336] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111)
I20250905 02:22:07.845070 20122 leader_election.cc:290] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857), 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697)
I20250905 02:22:07.852257 20127 raft_consensus.cc:513] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:07.856367 20050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8f259ed87244e51b1e3c174e81cea31" candidate_uuid: "6850db245d904b3e977ffc92f91c5c4d" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ec017dec7c9b42128e05beaa6f879077"
I20250905 02:22:07.857173 20050 raft_consensus.cc:3058] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:07.866832 20050 raft_consensus.cc:2466] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6850db245d904b3e977ffc92f91c5c4d in term 3.
W20250905 02:22:07.875504 19837 leader_election.cc:336] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111)
I20250905 02:22:07.876577 20127 leader_election.cc:290] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 election: Requested vote from peers ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857), 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697)
I20250905 02:22:07.878400 19839 leader_election.cc:304] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6850db245d904b3e977ffc92f91c5c4d, ec017dec7c9b42128e05beaa6f879077; no voters: 2057551e7a3b4d259fc6f55d74fcdbac
I20250905 02:22:07.879992 20127 raft_consensus.cc:2802] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:22:07.881647 20127 raft_consensus.cc:695] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [term 3 LEADER]: Becoming Leader. State: Replica: 6850db245d904b3e977ffc92f91c5c4d, State: Running, Role: LEADER
I20250905 02:22:07.883327 20127 consensus_queue.cc:237] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:07.903669 19761 catalog_manager.cc:5582] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250905 02:22:08.017338 19837 consensus_peers.cc:489] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d -> Peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Couldn't send request to peer 2057551e7a3b4d259fc6f55d74fcdbac. Status: Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:08.244823 20050 raft_consensus.cc:1273] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Refusing update from remote peer 6850db245d904b3e977ffc92f91c5c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250905 02:22:08.247339 20127 consensus_queue.cc:1035] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250905 02:22:08.287438 20048 raft_consensus.cc:1273] T b8f259ed87244e51b1e3c174e81cea31 P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Refusing update from remote peer 6850db245d904b3e977ffc92f91c5c4d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250905 02:22:08.289469 20122 consensus_queue.cc:1035] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20250905 02:22:08.401984 19837 consensus_peers.cc:489] T b8f259ed87244e51b1e3c174e81cea31 P 6850db245d904b3e977ffc92f91c5c4d -> Peer 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): Couldn't send request to peer 2057551e7a3b4d259fc6f55d74fcdbac. Status: Network error: Client connection negotiation failed: client connection to 127.15.105.3:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:08.415329 19905 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:08.421937 20049 raft_consensus.cc:1273] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Refusing update from remote peer 6850db245d904b3e977ffc92f91c5c4d: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250905 02:22:08.423550 20138 consensus_queue.cc:1035] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250905 02:22:08.430471 20138 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:22:08.440085 20049 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } }
I20250905 02:22:08.457403 19748 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3292527771b441a49fce17c4d90f3a92 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:22:08.467862 19761 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: config changed from index 11 to 13, VOTER 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3) evicted. New cstate: current_term: 3 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 02:22:08.511442 19761 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 3292527771b441a49fce17c4d90f3a92 on TS 2057551e7a3b4d259fc6f55d74fcdbac: Not found: failed to reset TS proxy: Could not find TS for UUID 2057551e7a3b4d259fc6f55d74fcdbac
I20250905 02:22:08.520892 19905 consensus_queue.cc:237] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } }
I20250905 02:22:08.525856 20138 raft_consensus.cc:2953] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } }
I20250905 02:22:08.536897 19748 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3292527771b441a49fce17c4d90f3a92 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:22:08.541505 19762 catalog_manager.cc:5582] T 3292527771b441a49fce17c4d90f3a92 P 6850db245d904b3e977ffc92f91c5c4d reported cstate change: config changed from index 13 to 14, VOTER ec017dec7c9b42128e05beaa6f879077 (127.15.105.2) evicted. New cstate: current_term: 3 leader_uuid: "6850db245d904b3e977ffc92f91c5c4d" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 02:22:08.568532 19747 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 3292527771b441a49fce17c4d90f3a92 on TS 2057551e7a3b4d259fc6f55d74fcdbac failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2057551e7a3b4d259fc6f55d74fcdbac
I20250905 02:22:08.585708 20030 tablet_service.cc:1515] Processing DeleteTablet for tablet 3292527771b441a49fce17c4d90f3a92 with delete_type TABLET_DATA_TOMBSTONED (TS ec017dec7c9b42128e05beaa6f879077 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:53108
I20250905 02:22:08.598557 20148 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:08.599606 20148 raft_consensus.cc:2241] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:08.600335 20148 raft_consensus.cc:2270] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:08.603739 20148 ts_tablet_manager.cc:1905] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:22:08.622609 20148 ts_tablet_manager.cc:1918] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250905 02:22:08.623092 20148 log.cc:1199] T 3292527771b441a49fce17c4d90f3a92 P ec017dec7c9b42128e05beaa6f879077: Deleting WAL directory at /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/wals/3292527771b441a49fce17c4d90f3a92
I20250905 02:22:08.625218 19749 catalog_manager.cc:4928] TS ec017dec7c9b42128e05beaa6f879077 (127.15.105.2:45857): tablet 3292527771b441a49fce17c4d90f3a92 (table TestTable [id=b569ba47a7fd47fcb1f4d475b0959129]) successfully deleted
W20250905 02:22:06.976708 20117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:08.379249 20115 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 20099
W20250905 02:22:08.741266 20115 kernel_stack_watchdog.cc:198] Thread 20099 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:22:08.742081 20118 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1764 milliseconds
W20250905 02:22:08.741534 20099 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.766s user 0.632s sys 1.073s
W20250905 02:22:08.742667 20099 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.767s user 0.632s sys 1.073s
I20250905 02:22:08.743314 20099 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 02:22:08.743360 20119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:08.746773 20099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:08.748785 20099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:08.750114 20099 hybrid_clock.cc:648] HybridClock initialized: now 1757038928750069 us; error 51 us; skew 500 ppm
I20250905 02:22:08.750864 20099 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:08.756702 20099 webserver.cc:480] Webserver started at http://127.15.105.3:46845/ using document root <none> and password file <none>
I20250905 02:22:08.757584 20099 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:08.757793 20099 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:08.765339 20099 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250905 02:22:08.769941 20153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:08.770933 20099 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250905 02:22:08.771263 20099 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2057551e7a3b4d259fc6f55d74fcdbac"
format_stamp: "Formatted at 2025-09-05 02:21:37 on dist-test-slave-ctw4"
I20250905 02:22:08.773164 20099 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:08.822896 20099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:08.824426 20099 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:08.824857 20099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:08.827263 20099 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:08.833482 20160 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250905 02:22:08.845489 20099 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250905 02:22:08.845714 20099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.000s
I20250905 02:22:08.846010 20099 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250905 02:22:08.851017 20160 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap starting.
I20250905 02:22:08.853598 20099 ts_tablet_manager.cc:610] Registered 2 tablets
I20250905 02:22:08.853777 20099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20250905 02:22:08.906565 20160 log.cc:826] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:09.026160 20099 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:37697
I20250905 02:22:09.026268 20267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:37697 every 8 connection(s)
I20250905 02:22:09.029603 20099 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 02:22:09.038434 20160 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:09.039237 20160 tablet_bootstrap.cc:492] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap complete.
I20250905 02:22:09.039462 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 20099
I20250905 02:22:09.040584 20160 ts_tablet_manager.cc:1397] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent bootstrapping tablet: real 0.190s user 0.136s sys 0.051s
I20250905 02:22:09.058084 20160 raft_consensus.cc:357] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:09.061061 20160 raft_consensus.cc:738] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057551e7a3b4d259fc6f55d74fcdbac, State: Initialized, Role: FOLLOWER
I20250905 02:22:09.061499 20268 heartbeater.cc:344] Connected to a master server at 127.15.105.62:41029
I20250905 02:22:09.061935 20268 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:09.061954 20160 consensus_queue.cc:260] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } } peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } attrs { promote: false } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } }
I20250905 02:22:09.063323 20268 heartbeater.cc:507] Master 127.15.105.62:41029 requested a full tablet report, sending...
I20250905 02:22:09.065404 20160 ts_tablet_manager.cc:1428] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent starting tablet: real 0.025s user 0.018s sys 0.005s
I20250905 02:22:09.066020 20160 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap starting.
I20250905 02:22:09.067140 19762 ts_manager.cc:194] Registered new tserver with Master: 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697)
I20250905 02:22:09.071516 19762 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:60999
I20250905 02:22:09.075070 20268 heartbeater.cc:499] Master 127.15.105.62:41029 was elected leader, sending a full tablet report...
I20250905 02:22:09.078670 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:22:09.082945 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250905 02:22:09.085821 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250905 02:22:09.154121 20160 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:09.154816 20160 tablet_bootstrap.cc:492] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Bootstrap complete.
I20250905 02:22:09.155965 20160 ts_tablet_manager.cc:1397] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent bootstrapping tablet: real 0.090s user 0.082s sys 0.007s
I20250905 02:22:09.157503 20160 raft_consensus.cc:357] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:09.157932 20160 raft_consensus.cc:738] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057551e7a3b4d259fc6f55d74fcdbac, State: Initialized, Role: FOLLOWER
I20250905 02:22:09.158424 20160 consensus_queue.cc:260] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6850db245d904b3e977ffc92f91c5c4d" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 39371 } } peers { permanent_uuid: "ec017dec7c9b42128e05beaa6f879077" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 45857 } attrs { promote: false } } peers { permanent_uuid: "2057551e7a3b4d259fc6f55d74fcdbac" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 37697 } attrs { promote: false } }
I20250905 02:22:09.159930 20160 ts_tablet_manager.cc:1428] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 02:22:09.192283 20203 tablet_service.cc:1515] Processing DeleteTablet for tablet 3292527771b441a49fce17c4d90f3a92 with delete_type TABLET_DATA_TOMBSTONED (TS 2057551e7a3b4d259fc6f55d74fcdbac not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:34038
I20250905 02:22:09.197077 20278 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:09.197679 20278 raft_consensus.cc:2241] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:09.198064 20278 raft_consensus.cc:2270] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:09.200223 20278 ts_tablet_manager.cc:1905] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:22:09.211637 20278 ts_tablet_manager.cc:1918] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250905 02:22:09.211959 20278 log.cc:1199] T 3292527771b441a49fce17c4d90f3a92 P 2057551e7a3b4d259fc6f55d74fcdbac: Deleting WAL directory at /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/wals/3292527771b441a49fce17c4d90f3a92
I20250905 02:22:09.213584 19747 catalog_manager.cc:4928] TS 2057551e7a3b4d259fc6f55d74fcdbac (127.15.105.3:37697): tablet 3292527771b441a49fce17c4d90f3a92 (table TestTable [id=b569ba47a7fd47fcb1f4d475b0959129]) successfully deleted
I20250905 02:22:09.299696 20223 raft_consensus.cc:3058] T b8f259ed87244e51b1e3c174e81cea31 P 2057551e7a3b4d259fc6f55d74fcdbac [term 2 FOLLOWER]: Advancing to term 3
W20250905 02:22:10.089985 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:11.093377 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:12.096630 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:13.100343 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:14.104025 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:15.107098 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:16.110248 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:17.113478 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:18.116588 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:19.119621 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:20.122797 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:21.125803 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:22.128857 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:23.132719 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:23.535710 20264 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdca0 after lost signal to thread 20111
W20250905 02:22:23.536509 20264 debug-util.cc:398] Leaking SignalData structure 0x7b08000cd7c0 after lost signal to thread 20267
W20250905 02:22:24.136197 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:25.139320 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:26.142377 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:27.145560 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 02:22:28.148731 15780 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3292527771b441a49fce17c4d90f3a92: tablet_id: "3292527771b441a49fce17c4d90f3a92" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250905 02:22:29.151535 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 19799
I20250905 02:22:29.177083 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 19951
I20250905 02:22:29.203507 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 20099
I20250905 02:22:29.229043 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 19729
2025-09-05T02:22:29Z chronyd exiting
I20250905 02:22:29.275825 15780 test_util.cc:183] -----------------------------------------------
I20250905 02:22:29.276038 15780 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757038821926622-15780-0
[ FAILED ] AdminCliTest.TestRebuildTables (59566 ms)
[----------] 5 tests from AdminCliTest (127263 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250905 02:22:29.279575 15780 test_util.cc:276] Using random seed: -2106742971
I20250905 02:22:29.283211 15780 ts_itest-base.cc:115] Starting cluster with:
I20250905 02:22:29.283332 15780 ts_itest-base.cc:116] --------------
I20250905 02:22:29.283470 15780 ts_itest-base.cc:117] 5 tablet servers
I20250905 02:22:29.283588 15780 ts_itest-base.cc:118] 3 replicas per TS
I20250905 02:22:29.283701 15780 ts_itest-base.cc:119] --------------
2025-09-05T02:22:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T02:22:29Z Disabled control of system clock
I20250905 02:22:29.323318 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:38511
--webserver_interface=127.15.105.62
--webserver_port=0
--builtin_ntp_servers=127.15.105.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:38511
--raft_prepare_replacement_before_eviction=true with env {}
W20250905 02:22:29.601398 20305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:29.601910 20305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:29.602270 20305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:29.630491 20305 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 02:22:29.630833 20305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:22:29.631053 20305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:29.631222 20305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:22:29.631388 20305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:22:29.663412 20305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:38511
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:38511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:29.664686 20305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:29.666123 20305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:29.676614 20311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:31.080024 20310 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 20305
W20250905 02:22:31.403563 20305 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.727s user 0.608s sys 1.119s
W20250905 02:22:29.676955 20312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:31.403995 20305 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.728s user 0.608s sys 1.120s
W20250905 02:22:31.405578 20314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:31.408432 20313 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1728 milliseconds
I20250905 02:22:31.408510 20305 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:31.409597 20305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:31.411923 20305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:31.413244 20305 hybrid_clock.cc:648] HybridClock initialized: now 1757038951413194 us; error 57 us; skew 500 ppm
I20250905 02:22:31.414000 20305 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:31.419864 20305 webserver.cc:480] Webserver started at http://127.15.105.62:45149/ using document root <none> and password file <none>
I20250905 02:22:31.420717 20305 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:31.420917 20305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:31.421334 20305 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:31.425580 20305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1a42c61216114194b6765172365eae4f"
format_stamp: "Formatted at 2025-09-05 02:22:31 on dist-test-slave-ctw4"
I20250905 02:22:31.426657 20305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1a42c61216114194b6765172365eae4f"
format_stamp: "Formatted at 2025-09-05 02:22:31 on dist-test-slave-ctw4"
I20250905 02:22:31.433470 20305 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250905 02:22:31.438618 20321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:31.439579 20305 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250905 02:22:31.439917 20305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1a42c61216114194b6765172365eae4f"
format_stamp: "Formatted at 2025-09-05 02:22:31 on dist-test-slave-ctw4"
I20250905 02:22:31.440233 20305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:31.483863 20305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:31.485144 20305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:31.485532 20305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:31.551322 20305 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:38511
I20250905 02:22:31.551390 20372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:38511 every 8 connection(s)
I20250905 02:22:31.553900 20305 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:22:31.555167 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 20305
I20250905 02:22:31.555625 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 02:22:31.561018 20373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:31.584159 20373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f: Bootstrap starting.
I20250905 02:22:31.589404 20373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:31.591243 20373 log.cc:826] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:31.595247 20373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f: No bootstrap required, opened a new log
I20250905 02:22:31.612598 20373 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a42c61216114194b6765172365eae4f" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 38511 } }
I20250905 02:22:31.613158 20373 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:31.613336 20373 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a42c61216114194b6765172365eae4f, State: Initialized, Role: FOLLOWER
I20250905 02:22:31.613930 20373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a42c61216114194b6765172365eae4f" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 38511 } }
I20250905 02:22:31.614367 20373 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:31.614589 20373 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:31.614840 20373 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:31.619344 20373 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a42c61216114194b6765172365eae4f" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 38511 } }
I20250905 02:22:31.619949 20373 leader_election.cc:304] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a42c61216114194b6765172365eae4f; no voters:
I20250905 02:22:31.621385 20373 leader_election.cc:290] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:22:31.622102 20378 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:31.624485 20378 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [term 1 LEADER]: Becoming Leader. State: Replica: 1a42c61216114194b6765172365eae4f, State: Running, Role: LEADER
I20250905 02:22:31.624933 20373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:22:31.625424 20378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a42c61216114194b6765172365eae4f" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 38511 } }
I20250905 02:22:31.634992 20380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a42c61216114194b6765172365eae4f. Latest consensus state: current_term: 1 leader_uuid: "1a42c61216114194b6765172365eae4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a42c61216114194b6765172365eae4f" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 38511 } } }
I20250905 02:22:31.635788 20380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:31.635497 20379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a42c61216114194b6765172365eae4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a42c61216114194b6765172365eae4f" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 38511 } } }
I20250905 02:22:31.636198 20379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:31.640169 20386 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:22:31.650964 20386 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:22:31.666983 20386 catalog_manager.cc:1349] Generated new cluster ID: dc9a36e4c44746d08d06d87920ada3fd
I20250905 02:22:31.667237 20386 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:22:31.693450 20386 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:22:31.694839 20386 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:22:31.707952 20386 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1a42c61216114194b6765172365eae4f: Generated new TSK 0
I20250905 02:22:31.708710 20386 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:22:31.728312 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:0
--local_ip_for_outbound_sockets=127.15.105.1
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--builtin_ntp_servers=127.15.105.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250905 02:22:32.032902 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:32.033358 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:32.033902 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:32.062255 20397 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 02:22:32.062600 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:32.063297 20397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:22:32.126092 20397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:32.127403 20397 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:32.129086 20397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:32.142989 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:33.544849 20402 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 20397
W20250905 02:22:32.144254 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:33.891052 20402 kernel_stack_watchdog.cc:198] Thread 20397 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:22:33.891661 20397 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.747s user 0.603s sys 1.094s
W20250905 02:22:33.892038 20397 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.748s user 0.603s sys 1.094s
W20250905 02:22:33.894124 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:33.898065 20405 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1752 milliseconds
I20250905 02:22:33.898066 20397 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:33.899255 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:33.901500 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:33.902822 20397 hybrid_clock.cc:648] HybridClock initialized: now 1757038953902791 us; error 37 us; skew 500 ppm
I20250905 02:22:33.903514 20397 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:33.909235 20397 webserver.cc:480] Webserver started at http://127.15.105.1:33439/ using document root <none> and password file <none>
I20250905 02:22:33.910161 20397 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:33.910375 20397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:33.910794 20397 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:33.914892 20397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "11f4c1d4b4a845a8bfafb6ca37702ed6"
format_stamp: "Formatted at 2025-09-05 02:22:33 on dist-test-slave-ctw4"
I20250905 02:22:33.915961 20397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "11f4c1d4b4a845a8bfafb6ca37702ed6"
format_stamp: "Formatted at 2025-09-05 02:22:33 on dist-test-slave-ctw4"
I20250905 02:22:33.922487 20397 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250905 02:22:33.927853 20414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:33.928897 20397 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250905 02:22:33.929183 20397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "11f4c1d4b4a845a8bfafb6ca37702ed6"
format_stamp: "Formatted at 2025-09-05 02:22:33 on dist-test-slave-ctw4"
I20250905 02:22:33.929486 20397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:33.975193 20397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:33.976565 20397 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:33.976975 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:33.979815 20397 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:33.983470 20397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:33.983647 20397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:33.983877 20397 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:33.984014 20397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:34.135897 20397 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:45011
I20250905 02:22:34.136073 20526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:45011 every 8 connection(s)
I20250905 02:22:34.138387 20397 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:22:34.145135 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 20397
I20250905 02:22:34.145864 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 02:22:34.154174 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:0
--local_ip_for_outbound_sockets=127.15.105.2
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--builtin_ntp_servers=127.15.105.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250905 02:22:34.161309 20527 heartbeater.cc:344] Connected to a master server at 127.15.105.62:38511
I20250905 02:22:34.161777 20527 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:34.162786 20527 heartbeater.cc:507] Master 127.15.105.62:38511 requested a full tablet report, sending...
I20250905 02:22:34.165513 20338 ts_manager.cc:194] Registered new tserver with Master: 11f4c1d4b4a845a8bfafb6ca37702ed6 (127.15.105.1:45011)
I20250905 02:22:34.168188 20338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:46149
W20250905 02:22:34.448010 20531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:34.448498 20531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:34.448982 20531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:34.479454 20531 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 02:22:34.479877 20531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:34.480612 20531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:22:34.512876 20531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:34.514072 20531 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:34.515537 20531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:34.526142 20537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:35.172233 20527 heartbeater.cc:499] Master 127.15.105.62:38511 was elected leader, sending a full tablet report...
W20250905 02:22:35.930441 20536 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 20531
W20250905 02:22:36.217615 20531 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.691s user 0.702s sys 0.987s
W20250905 02:22:36.219565 20531 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.693s user 0.703s sys 0.987s
W20250905 02:22:34.526698 20538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:36.219986 20531 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 02:22:36.220106 20539 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1692 milliseconds
W20250905 02:22:36.220057 20540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:36.223764 20531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:36.225638 20531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:36.226939 20531 hybrid_clock.cc:648] HybridClock initialized: now 1757038956226902 us; error 44 us; skew 500 ppm
I20250905 02:22:36.227658 20531 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:36.233626 20531 webserver.cc:480] Webserver started at http://127.15.105.2:38233/ using document root <none> and password file <none>
I20250905 02:22:36.234489 20531 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:36.234671 20531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:36.235049 20531 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:36.239183 20531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "afba28b856f74bceaad2111855354604"
format_stamp: "Formatted at 2025-09-05 02:22:36 on dist-test-slave-ctw4"
I20250905 02:22:36.240226 20531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "afba28b856f74bceaad2111855354604"
format_stamp: "Formatted at 2025-09-05 02:22:36 on dist-test-slave-ctw4"
I20250905 02:22:36.247282 20531 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250905 02:22:36.252496 20547 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:36.253409 20531 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250905 02:22:36.253756 20531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "afba28b856f74bceaad2111855354604"
format_stamp: "Formatted at 2025-09-05 02:22:36 on dist-test-slave-ctw4"
I20250905 02:22:36.254132 20531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:36.310349 20531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:36.311703 20531 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:36.312119 20531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:36.314474 20531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:36.318351 20531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:36.318555 20531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:36.318783 20531 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:36.318922 20531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:36.446537 20531 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:40709
I20250905 02:22:36.446630 20659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:40709 every 8 connection(s)
I20250905 02:22:36.449112 20531 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 02:22:36.458608 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 20531
I20250905 02:22:36.459137 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 02:22:36.466010 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:0
--local_ip_for_outbound_sockets=127.15.105.3
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--builtin_ntp_servers=127.15.105.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250905 02:22:36.472069 20660 heartbeater.cc:344] Connected to a master server at 127.15.105.62:38511
I20250905 02:22:36.472484 20660 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:36.473446 20660 heartbeater.cc:507] Master 127.15.105.62:38511 requested a full tablet report, sending...
I20250905 02:22:36.475334 20338 ts_manager.cc:194] Registered new tserver with Master: afba28b856f74bceaad2111855354604 (127.15.105.2:40709)
I20250905 02:22:36.476533 20338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:50043
W20250905 02:22:36.752439 20664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:36.752938 20664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:36.753434 20664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:36.781538 20664 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 02:22:36.781912 20664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:36.782645 20664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:22:36.814160 20664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:36.815414 20664 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:36.816927 20664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:36.828868 20670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:37.479346 20660 heartbeater.cc:499] Master 127.15.105.62:38511 was elected leader, sending a full tablet report...
W20250905 02:22:36.829317 20671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:38.231657 20669 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 20664
W20250905 02:22:38.451751 20664 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.623s user 0.675s sys 0.948s
W20250905 02:22:38.453570 20664 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.625s user 0.675s sys 0.948s
W20250905 02:22:38.454098 20673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:38.457089 20672 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1623 milliseconds
I20250905 02:22:38.457134 20664 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:38.458334 20664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:38.460307 20664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:38.461660 20664 hybrid_clock.cc:648] HybridClock initialized: now 1757038958461613 us; error 55 us; skew 500 ppm
I20250905 02:22:38.462438 20664 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:38.468266 20664 webserver.cc:480] Webserver started at http://127.15.105.3:36921/ using document root <none> and password file <none>
I20250905 02:22:38.469132 20664 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:38.469372 20664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:38.469769 20664 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:38.473852 20664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8b00ada33b8b432bb02a3dc07562e007"
format_stamp: "Formatted at 2025-09-05 02:22:38 on dist-test-slave-ctw4"
I20250905 02:22:38.474815 20664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8b00ada33b8b432bb02a3dc07562e007"
format_stamp: "Formatted at 2025-09-05 02:22:38 on dist-test-slave-ctw4"
I20250905 02:22:38.481240 20664 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.006s
I20250905 02:22:38.486768 20680 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:38.487880 20664 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250905 02:22:38.488157 20664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8b00ada33b8b432bb02a3dc07562e007"
format_stamp: "Formatted at 2025-09-05 02:22:38 on dist-test-slave-ctw4"
I20250905 02:22:38.488440 20664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:38.533002 20664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:38.534243 20664 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:38.534605 20664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:38.536897 20664 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:38.540601 20664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:38.540786 20664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:38.540973 20664 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:38.541087 20664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:38.667088 20664 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:32979
I20250905 02:22:38.667176 20792 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:32979 every 8 connection(s)
I20250905 02:22:38.669533 20664 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 02:22:38.672497 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 20664
I20250905 02:22:38.672981 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 02:22:38.680027 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.4:0
--local_ip_for_outbound_sockets=127.15.105.4
--webserver_interface=127.15.105.4
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--builtin_ntp_servers=127.15.105.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250905 02:22:38.694031 20793 heartbeater.cc:344] Connected to a master server at 127.15.105.62:38511
I20250905 02:22:38.694499 20793 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:38.695741 20793 heartbeater.cc:507] Master 127.15.105.62:38511 requested a full tablet report, sending...
I20250905 02:22:38.698135 20338 ts_manager.cc:194] Registered new tserver with Master: 8b00ada33b8b432bb02a3dc07562e007 (127.15.105.3:32979)
I20250905 02:22:38.699608 20338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:52331
W20250905 02:22:38.967093 20797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:38.967607 20797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:38.968134 20797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:38.996927 20797 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 02:22:38.997313 20797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:38.998111 20797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.4
I20250905 02:22:39.029342 20797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.105.4
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.4
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:39.030575 20797 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:39.032111 20797 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:39.042200 20803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:39.703177 20793 heartbeater.cc:499] Master 127.15.105.62:38511 was elected leader, sending a full tablet report...
W20250905 02:22:39.043712 20804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:40.166288 20806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:40.169728 20805 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1125 milliseconds
I20250905 02:22:40.169854 20797 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:40.171056 20797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:40.173485 20797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:40.174908 20797 hybrid_clock.cc:648] HybridClock initialized: now 1757038960174854 us; error 54 us; skew 500 ppm
I20250905 02:22:40.175663 20797 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:40.181841 20797 webserver.cc:480] Webserver started at http://127.15.105.4:40409/ using document root <none> and password file <none>
I20250905 02:22:40.182744 20797 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:40.182955 20797 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:40.183374 20797 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:40.187537 20797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "62c225d7f17348bb9694902c2474314a"
format_stamp: "Formatted at 2025-09-05 02:22:40 on dist-test-slave-ctw4"
I20250905 02:22:40.188632 20797 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "62c225d7f17348bb9694902c2474314a"
format_stamp: "Formatted at 2025-09-05 02:22:40 on dist-test-slave-ctw4"
I20250905 02:22:40.195289 20797 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250905 02:22:40.200845 20814 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:40.201711 20797 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20250905 02:22:40.201967 20797 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "62c225d7f17348bb9694902c2474314a"
format_stamp: "Formatted at 2025-09-05 02:22:40 on dist-test-slave-ctw4"
I20250905 02:22:40.202250 20797 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:40.255177 20797 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:40.256681 20797 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:40.257036 20797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:40.259341 20797 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:40.262938 20797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:40.263116 20797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:40.263317 20797 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:40.263451 20797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250905 02:22:40.346200 20523 debug-util.cc:398] Leaking SignalData structure 0x7b08000acb20 after lost signal to thread 20398
W20250905 02:22:40.347401 20523 debug-util.cc:398] Leaking SignalData structure 0x7b08000b37c0 after lost signal to thread 20526
I20250905 02:22:40.397941 20797 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.4:44717
I20250905 02:22:40.397989 20926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.4:44717 every 8 connection(s)
I20250905 02:22:40.401811 20797 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250905 02:22:40.404151 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 20797
I20250905 02:22:40.404661 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250905 02:22:40.413220 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.5:0
--local_ip_for_outbound_sockets=127.15.105.5
--webserver_interface=127.15.105.5
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--builtin_ntp_servers=127.15.105.20:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250905 02:22:40.426514 20927 heartbeater.cc:344] Connected to a master server at 127.15.105.62:38511
I20250905 02:22:40.426996 20927 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:40.427999 20927 heartbeater.cc:507] Master 127.15.105.62:38511 requested a full tablet report, sending...
I20250905 02:22:40.430042 20338 ts_manager.cc:194] Registered new tserver with Master: 62c225d7f17348bb9694902c2474314a (127.15.105.4:44717)
I20250905 02:22:40.431363 20338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.4:43755
W20250905 02:22:40.738384 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:40.738880 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:40.739317 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:40.768837 20931 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 02:22:40.769208 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:40.769982 20931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.5
I20250905 02:22:40.802945 20931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.105.5
--webserver_port=0
--tserver_master_addrs=127.15.105.62:38511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.5
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:40.804273 20931 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:40.805817 20931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:40.817777 20937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:41.434329 20927 heartbeater.cc:499] Master 127.15.105.62:38511 was elected leader, sending a full tablet report...
W20250905 02:22:42.221006 20936 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 20931
W20250905 02:22:42.502784 20936 kernel_stack_watchdog.cc:198] Thread 20931 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:22:42.504856 20931 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.685s user 0.000s sys 0.002s
W20250905 02:22:42.505266 20931 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.686s user 0.000s sys 0.002s
W20250905 02:22:40.818863 20938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:42.506912 20940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:42.511380 20931 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 02:22:42.511379 20939 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1691 milliseconds
I20250905 02:22:42.512743 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:42.515255 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:42.516712 20931 hybrid_clock.cc:648] HybridClock initialized: now 1757038962516672 us; error 54 us; skew 500 ppm
I20250905 02:22:42.517486 20931 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:42.524319 20931 webserver.cc:480] Webserver started at http://127.15.105.5:37063/ using document root <none> and password file <none>
I20250905 02:22:42.525287 20931 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:42.525523 20931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:42.525945 20931 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:42.530364 20931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "bb4585f48b4343f994f05aafedd9c01f"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-ctw4"
I20250905 02:22:42.531481 20931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "bb4585f48b4343f994f05aafedd9c01f"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-ctw4"
I20250905 02:22:42.538975 20931 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250905 02:22:42.544868 20949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.545861 20931 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250905 02:22:42.546161 20931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "bb4585f48b4343f994f05aafedd9c01f"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-ctw4"
I20250905 02:22:42.546468 20931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:42.595749 20931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:42.597028 20931 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:42.597416 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:42.600253 20931 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:42.604470 20931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:42.604645 20931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.604888 20931 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:42.605084 20931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.777480 20931 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.5:44831
I20250905 02:22:42.777637 21061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.5:44831 every 8 connection(s)
I20250905 02:22:42.780138 20931 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250905 02:22:42.790277 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 20931
I20250905 02:22:42.791072 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250905 02:22:42.801862 21062 heartbeater.cc:344] Connected to a master server at 127.15.105.62:38511
I20250905 02:22:42.802222 21062 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:42.803539 21062 heartbeater.cc:507] Master 127.15.105.62:38511 requested a full tablet report, sending...
I20250905 02:22:42.805557 20338 ts_manager.cc:194] Registered new tserver with Master: bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831)
I20250905 02:22:42.806699 20338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.5:39245
I20250905 02:22:42.811951 15780 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250905 02:22:42.847680 20338 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43656:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250905 02:22:42.929603 20997 tablet_service.cc:1468] Processing CreateTablet for tablet bff6d768444e48b0a37cf3322c456bdb (DEFAULT_TABLE table=TestTable [id=2e6b600559984d378cc41c0e3ed04ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:22:42.932966 20997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff6d768444e48b0a37cf3322c456bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:42.934651 20595 tablet_service.cc:1468] Processing CreateTablet for tablet bff6d768444e48b0a37cf3322c456bdb (DEFAULT_TABLE table=TestTable [id=2e6b600559984d378cc41c0e3ed04ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:22:42.934965 20862 tablet_service.cc:1468] Processing CreateTablet for tablet bff6d768444e48b0a37cf3322c456bdb (DEFAULT_TABLE table=TestTable [id=2e6b600559984d378cc41c0e3ed04ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:22:42.936280 20862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff6d768444e48b0a37cf3322c456bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:42.936288 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bff6d768444e48b0a37cf3322c456bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:42.957211 21081 tablet_bootstrap.cc:492] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a: Bootstrap starting.
I20250905 02:22:42.975024 21082 tablet_bootstrap.cc:492] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f: Bootstrap starting.
I20250905 02:22:42.977136 21081 tablet_bootstrap.cc:654] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:42.980381 21081 log.cc:826] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:42.987618 21081 tablet_bootstrap.cc:492] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a: No bootstrap required, opened a new log
I20250905 02:22:42.988248 21081 ts_tablet_manager.cc:1397] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.015s
I20250905 02:22:42.992146 21082 tablet_bootstrap.cc:654] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:42.994628 21082 log.cc:826] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:43.002957 21082 tablet_bootstrap.cc:492] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f: No bootstrap required, opened a new log
I20250905 02:22:43.003414 21082 ts_tablet_manager.cc:1397] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.009s
I20250905 02:22:43.015516 21081 raft_consensus.cc:357] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.016513 21081 raft_consensus.cc:383] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:43.016872 21081 raft_consensus.cc:738] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c225d7f17348bb9694902c2474314a, State: Initialized, Role: FOLLOWER
I20250905 02:22:43.017890 21081 consensus_queue.cc:260] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.025920 21081 ts_tablet_manager.cc:1428] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a: Time spent starting tablet: real 0.037s user 0.027s sys 0.008s
I20250905 02:22:43.030545 21082 raft_consensus.cc:357] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.031167 21082 raft_consensus.cc:383] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:43.031383 21082 raft_consensus.cc:738] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb4585f48b4343f994f05aafedd9c01f, State: Initialized, Role: FOLLOWER
I20250905 02:22:43.032051 21082 consensus_queue.cc:260] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.035382 21062 heartbeater.cc:499] Master 127.15.105.62:38511 was elected leader, sending a full tablet report...
W20250905 02:22:43.037608 21063 tablet.cc:2378] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:22:43.040405 21082 ts_tablet_manager.cc:1428] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f: Time spent starting tablet: real 0.037s user 0.031s sys 0.000s
I20250905 02:22:43.041795 21087 tablet_bootstrap.cc:492] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604: Bootstrap starting.
I20250905 02:22:43.048547 21087 tablet_bootstrap.cc:654] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:43.050936 21087 log.cc:826] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:43.057404 21087 tablet_bootstrap.cc:492] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604: No bootstrap required, opened a new log
I20250905 02:22:43.057899 21087 ts_tablet_manager.cc:1397] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.001s
I20250905 02:22:43.071540 21086 raft_consensus.cc:491] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:43.072165 21086 raft_consensus.cc:513] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.075021 21086 leader_election.cc:290] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afba28b856f74bceaad2111855354604 (127.15.105.2:40709), 62c225d7f17348bb9694902c2474314a (127.15.105.4:44717)
I20250905 02:22:43.085193 21087 raft_consensus.cc:357] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.086015 21087 raft_consensus.cc:383] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:43.086282 21087 raft_consensus.cc:738] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afba28b856f74bceaad2111855354604, State: Initialized, Role: FOLLOWER
I20250905 02:22:43.087251 21087 consensus_queue.cc:260] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.091166 20882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "bb4585f48b4343f994f05aafedd9c01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c225d7f17348bb9694902c2474314a" is_pre_election: true
I20250905 02:22:43.091892 20882 raft_consensus.cc:2466] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb4585f48b4343f994f05aafedd9c01f in term 0.
I20250905 02:22:43.091892 20615 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "bb4585f48b4343f994f05aafedd9c01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afba28b856f74bceaad2111855354604" is_pre_election: true
I20250905 02:22:43.092620 20615 raft_consensus.cc:2466] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb4585f48b4343f994f05aafedd9c01f in term 0.
I20250905 02:22:43.093098 20953 leader_election.cc:304] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c225d7f17348bb9694902c2474314a, bb4585f48b4343f994f05aafedd9c01f; no voters:
I20250905 02:22:43.093557 21087 ts_tablet_manager.cc:1428] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604: Time spent starting tablet: real 0.035s user 0.033s sys 0.002s
I20250905 02:22:43.093954 21086 raft_consensus.cc:2802] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:43.094297 21086 raft_consensus.cc:491] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:43.094612 21086 raft_consensus.cc:3058] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:43.100688 21086 raft_consensus.cc:513] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.101977 21086 leader_election.cc:290] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [CANDIDATE]: Term 1 election: Requested vote from peers afba28b856f74bceaad2111855354604 (127.15.105.2:40709), 62c225d7f17348bb9694902c2474314a (127.15.105.4:44717)
I20250905 02:22:43.102875 20882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "bb4585f48b4343f994f05aafedd9c01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c225d7f17348bb9694902c2474314a"
I20250905 02:22:43.103062 20615 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "bb4585f48b4343f994f05aafedd9c01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afba28b856f74bceaad2111855354604"
I20250905 02:22:43.103372 20882 raft_consensus.cc:3058] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:43.103497 20615 raft_consensus.cc:3058] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:43.110512 20882 raft_consensus.cc:2466] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb4585f48b4343f994f05aafedd9c01f in term 1.
I20250905 02:22:43.111599 20953 leader_election.cc:304] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c225d7f17348bb9694902c2474314a, bb4585f48b4343f994f05aafedd9c01f; no voters:
I20250905 02:22:43.112218 21086 raft_consensus.cc:2802] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:43.112218 20615 raft_consensus.cc:2466] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb4585f48b4343f994f05aafedd9c01f in term 1.
I20250905 02:22:43.116401 21086 raft_consensus.cc:695] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [term 1 LEADER]: Becoming Leader. State: Replica: bb4585f48b4343f994f05aafedd9c01f, State: Running, Role: LEADER
I20250905 02:22:43.117201 21086 consensus_queue.cc:237] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:43.128211 20338 catalog_manager.cc:5582] T bff6d768444e48b0a37cf3322c456bdb P bb4585f48b4343f994f05aafedd9c01f reported cstate change: term changed from 0 to 1, leader changed from <none> to bb4585f48b4343f994f05aafedd9c01f (127.15.105.5). New cstate: current_term: 1 leader_uuid: "bb4585f48b4343f994f05aafedd9c01f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } health_report { overall_health: UNKNOWN } } }
W20250905 02:22:43.164270 20928 tablet.cc:2378] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:22:43.175372 15780 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250905 02:22:43.178478 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver afba28b856f74bceaad2111855354604 to finish bootstrapping
I20250905 02:22:43.191540 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 62c225d7f17348bb9694902c2474314a to finish bootstrapping
I20250905 02:22:43.201715 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bb4585f48b4343f994f05aafedd9c01f to finish bootstrapping
W20250905 02:22:43.211913 20661 tablet.cc:2378] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:22:43.212733 15780 test_util.cc:276] Using random seed: -2092809823
I20250905 02:22:43.236158 15780 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250905 02:22:43.236984 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 20931
W20250905 02:22:43.269622 21098 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 02:22:43.247177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.105.5:44831 (local address 127.0.0.1:42768)
0905 02:22:43.248078 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0905 02:22:43.248129 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0905 02:22:43.248407 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0905 02:22:43.259896 (+ 11489us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":68}
W20250905 02:22:43.271654 21094 meta_cache.cc:302] tablet bff6d768444e48b0a37cf3322c456bdb: replica bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831) has failed: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 02:22:43.292538 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.308081 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.328313 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.337164 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.362756 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.374707 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.404186 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.422154 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.458823 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.475929 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.520197 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.541033 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.592583 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.619217 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.672328 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.700556 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.769295 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.802927 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.875857 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:43.911339 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:43.990010 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:44.022722 20656 debug-util.cc:398] Leaking SignalData structure 0x7b080003c1a0 after lost signal to thread 20532
W20250905 02:22:44.031265 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:44.115736 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:44.157943 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:44.247256 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:44.291710 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:44.343950 21094 meta_cache.cc:302] tablet bff6d768444e48b0a37cf3322c456bdb: replica bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831) has failed: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
W20250905 02:22:44.390950 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:44.437812 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:44.548467 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:44.599004 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:44.709945 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:44.764639 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:44.876559 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:44.938993 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
I20250905 02:22:44.947871 21113 raft_consensus.cc:491] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:44.948638 21113 raft_consensus.cc:513] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:44.928514 21112 raft_consensus.cc:491] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:44.963641 21112 raft_consensus.cc:513] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:44.988715 21113 leader_election.cc:290] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers afba28b856f74bceaad2111855354604 (127.15.105.2:40709), bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831)
W20250905 02:22:44.992580 20815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
I20250905 02:22:44.969529 21112 leader_election.cc:290] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831), 62c225d7f17348bb9694902c2474314a (127.15.105.4:44717)
W20250905 02:22:45.021582 20815 leader_election.cc:336] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831): Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
W20250905 02:22:45.026563 20548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
W20250905 02:22:45.038707 20548 leader_election.cc:336] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831): Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
I20250905 02:22:45.040429 20615 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "62c225d7f17348bb9694902c2474314a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afba28b856f74bceaad2111855354604" is_pre_election: true
I20250905 02:22:45.041148 20615 raft_consensus.cc:2466] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c225d7f17348bb9694902c2474314a in term 1.
I20250905 02:22:45.042232 20882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "afba28b856f74bceaad2111855354604" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c225d7f17348bb9694902c2474314a" is_pre_election: true
I20250905 02:22:45.042753 20882 raft_consensus.cc:2466] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afba28b856f74bceaad2111855354604 in term 1.
I20250905 02:22:45.044128 20818 leader_election.cc:304] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62c225d7f17348bb9694902c2474314a, afba28b856f74bceaad2111855354604; no voters: bb4585f48b4343f994f05aafedd9c01f
I20250905 02:22:45.045063 20551 leader_election.cc:304] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62c225d7f17348bb9694902c2474314a, afba28b856f74bceaad2111855354604; no voters: bb4585f48b4343f994f05aafedd9c01f
I20250905 02:22:45.046304 21112 raft_consensus.cc:2802] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:45.046648 21112 raft_consensus.cc:491] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:45.046950 21112 raft_consensus.cc:3058] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:45.054778 21112 raft_consensus.cc:513] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:45.055810 21113 raft_consensus.cc:2802] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:45.056169 21113 raft_consensus.cc:491] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:45.056442 21113 raft_consensus.cc:3058] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:45.063920 21113 raft_consensus.cc:513] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
W20250905 02:22:45.057528 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
I20250905 02:22:45.066598 20615 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "62c225d7f17348bb9694902c2474314a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afba28b856f74bceaad2111855354604"
I20250905 02:22:45.068424 20615 raft_consensus.cc:2391] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 62c225d7f17348bb9694902c2474314a in current term 2: Already voted for candidate afba28b856f74bceaad2111855354604 in this term.
W20250905 02:22:45.078794 20815 leader_election.cc:336] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831): Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
I20250905 02:22:45.079267 20815 leader_election.cc:304] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62c225d7f17348bb9694902c2474314a; no voters: afba28b856f74bceaad2111855354604, bb4585f48b4343f994f05aafedd9c01f
I20250905 02:22:45.082952 21113 leader_election.cc:290] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 2 election: Requested vote from peers afba28b856f74bceaad2111855354604 (127.15.105.2:40709), bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831)
W20250905 02:22:45.088418 20548 leader_election.cc:336] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831): Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
I20250905 02:22:45.088877 21122 raft_consensus.cc:2747] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 02:22:45.091127 20882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "afba28b856f74bceaad2111855354604" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c225d7f17348bb9694902c2474314a"
I20250905 02:22:45.092075 20882 raft_consensus.cc:2391] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate afba28b856f74bceaad2111855354604 in current term 2: Already voted for candidate 62c225d7f17348bb9694902c2474314a in this term.
I20250905 02:22:45.093120 20551 leader_election.cc:304] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afba28b856f74bceaad2111855354604; no voters: 62c225d7f17348bb9694902c2474314a, bb4585f48b4343f994f05aafedd9c01f
I20250905 02:22:45.098584 21112 leader_election.cc:290] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [CANDIDATE]: Term 2 election: Requested vote from peers bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831), 62c225d7f17348bb9694902c2474314a (127.15.105.4:44717)
I20250905 02:22:45.099550 21112 raft_consensus.cc:2747] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250905 02:22:45.128798 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:45.252171 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:45.313573 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:45.382247 21094 meta_cache.cc:302] tablet bff6d768444e48b0a37cf3322c456bdb: replica bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831) has failed: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
W20250905 02:22:45.447912 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:45.511708 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
I20250905 02:22:45.592726 20862 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 02:22:45.625916 20462 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 02:22:45.627589 20728 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250905 02:22:45.651937 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
I20250905 02:22:45.676712 20595 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250905 02:22:45.721459 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:45.864032 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:45.935240 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:46.082376 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:46.155931 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:46.308250 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:46.384774 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:46.466637 21094 meta_cache.cc:302] tablet bff6d768444e48b0a37cf3322c456bdb: replica bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831) has failed: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
W20250905 02:22:46.545369 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
W20250905 02:22:46.624717 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Illegal state: replica 62c225d7f17348bb9694902c2474314a is not leader of this config: current role FOLLOWER
W20250905 02:22:46.785365 20575 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:50640: Illegal state: replica afba28b856f74bceaad2111855354604 is not leader of this config: current role FOLLOWER
I20250905 02:22:46.840184 21151 raft_consensus.cc:491] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:46.840585 21151 raft_consensus.cc:513] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:46.841977 21151 leader_election.cc:290] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers afba28b856f74bceaad2111855354604 (127.15.105.2:40709), bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831)
I20250905 02:22:46.842907 20615 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "62c225d7f17348bb9694902c2474314a" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afba28b856f74bceaad2111855354604" is_pre_election: true
I20250905 02:22:46.843500 20615 raft_consensus.cc:2466] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c225d7f17348bb9694902c2474314a in term 2.
I20250905 02:22:46.844590 20818 leader_election.cc:304] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c225d7f17348bb9694902c2474314a, afba28b856f74bceaad2111855354604; no voters:
I20250905 02:22:46.845466 21151 raft_consensus.cc:2802] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 02:22:46.845798 21151 raft_consensus.cc:491] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:46.846127 21151 raft_consensus.cc:3058] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 2 FOLLOWER]: Advancing to term 3
W20250905 02:22:46.847416 20815 leader_election.cc:336] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831): Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
I20250905 02:22:46.850689 21151 raft_consensus.cc:513] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
I20250905 02:22:46.851994 21151 leader_election.cc:290] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 3 election: Requested vote from peers afba28b856f74bceaad2111855354604 (127.15.105.2:40709), bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831)
I20250905 02:22:46.852821 20615 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bff6d768444e48b0a37cf3322c456bdb" candidate_uuid: "62c225d7f17348bb9694902c2474314a" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afba28b856f74bceaad2111855354604"
I20250905 02:22:46.853319 20615 raft_consensus.cc:3058] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 2 FOLLOWER]: Advancing to term 3
W20250905 02:22:46.854928 20815 leader_election.cc:336] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831): Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111)
I20250905 02:22:46.859702 20615 raft_consensus.cc:2466] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c225d7f17348bb9694902c2474314a in term 3.
I20250905 02:22:46.860626 20818 leader_election.cc:304] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62c225d7f17348bb9694902c2474314a, afba28b856f74bceaad2111855354604; no voters: bb4585f48b4343f994f05aafedd9c01f
I20250905 02:22:46.861325 21151 raft_consensus.cc:2802] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:22:46.862689 21151 raft_consensus.cc:695] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [term 3 LEADER]: Becoming Leader. State: Replica: 62c225d7f17348bb9694902c2474314a, State: Running, Role: LEADER
I20250905 02:22:46.863490 21151 consensus_queue.cc:237] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } }
W20250905 02:22:46.865715 20842 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:38346: Service unavailable: leader is not yet ready
I20250905 02:22:46.873797 20337 catalog_manager.cc:5582] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a reported cstate change: term changed from 1 to 3, leader changed from bb4585f48b4343f994f05aafedd9c01f (127.15.105.5) to 62c225d7f17348bb9694902c2474314a (127.15.105.4). New cstate: current_term: 3 leader_uuid: "62c225d7f17348bb9694902c2474314a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb4585f48b4343f994f05aafedd9c01f" member_type: VOTER last_known_addr { host: "127.15.105.5" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c225d7f17348bb9694902c2474314a" member_type: VOTER last_known_addr { host: "127.15.105.4" port: 44717 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:46.954526 20615 raft_consensus.cc:1273] T bff6d768444e48b0a37cf3322c456bdb P afba28b856f74bceaad2111855354604 [term 3 FOLLOWER]: Refusing update from remote peer 62c225d7f17348bb9694902c2474314a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 02:22:46.955688 21154 consensus_queue.cc:1035] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a [LEADER]: Connected to new peer: Peer: permanent_uuid: "afba28b856f74bceaad2111855354604" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:22:46.956197 20815 consensus_peers.cc:489] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a -> Peer bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831): Couldn't send request to peer bb4585f48b4343f994f05aafedd9c01f. Status: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:47.338759 20862 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 02:22:47.353222 20462 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 02:22:47.376454 20728 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 02:22:47.379456 20595 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250905 02:22:49.268982 20815 consensus_peers.cc:489] T bff6d768444e48b0a37cf3322c456bdb P 62c225d7f17348bb9694902c2474314a -> Peer bb4585f48b4343f994f05aafedd9c01f (127.15.105.5:44831): Couldn't send request to peer bb4585f48b4343f994f05aafedd9c01f. Status: Network error: Client connection negotiation failed: client connection to 127.15.105.5:44831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250905 02:22:49.839363 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 20397
I20250905 02:22:49.865797 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 20531
I20250905 02:22:49.899262 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 20664
I20250905 02:22:49.921792 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 20797
I20250905 02:22:49.958063 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 20305
2025-09-05T02:22:49Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (20733 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (20734 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250905 02:22:50.014885 15780 test_util.cc:276] Using random seed: -2086007677
I20250905 02:22:50.019950 15780 ts_itest-base.cc:115] Starting cluster with:
I20250905 02:22:50.020125 15780 ts_itest-base.cc:116] --------------
I20250905 02:22:50.020294 15780 ts_itest-base.cc:117] 1 tablet servers
I20250905 02:22:50.020434 15780 ts_itest-base.cc:118] 1 replicas per TS
I20250905 02:22:50.020571 15780 ts_itest-base.cc:119] --------------
2025-09-05T02:22:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T02:22:50Z Disabled control of system clock
I20250905 02:22:50.064895 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:39469
--webserver_interface=127.15.105.62
--webserver_port=0
--builtin_ntp_servers=127.15.105.20:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:39469 with env {}
W20250905 02:22:50.366492 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:50.367069 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:50.367466 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:50.398772 21212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:22:50.399076 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:50.399276 21212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:22:50.399469 21212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:22:50.434985 21212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:39469
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:39469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:50.436203 21212 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:50.437721 21212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:50.449358 21218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:51.852383 21217 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 21212
W20250905 02:22:50.449810 21219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:52.254619 21217 kernel_stack_watchdog.cc:198] Thread 21212 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:22:52.255369 21212 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.806s user 0.634s sys 1.170s
W20250905 02:22:52.255749 21212 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.807s user 0.634s sys 1.170s
W20250905 02:22:52.257066 21221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:52.259577 21220 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1806 milliseconds
I20250905 02:22:52.259641 21212 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:52.260804 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:52.263010 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:52.264322 21212 hybrid_clock.cc:648] HybridClock initialized: now 1757038972264282 us; error 46 us; skew 500 ppm
I20250905 02:22:52.265053 21212 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:52.270852 21212 webserver.cc:480] Webserver started at http://127.15.105.62:36975/ using document root <none> and password file <none>
I20250905 02:22:52.271730 21212 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:52.272006 21212 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:52.272430 21212 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:52.276769 21212 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9efda4364798446e8af7e1b1fe6fabc8"
format_stamp: "Formatted at 2025-09-05 02:22:52 on dist-test-slave-ctw4"
I20250905 02:22:52.277812 21212 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9efda4364798446e8af7e1b1fe6fabc8"
format_stamp: "Formatted at 2025-09-05 02:22:52 on dist-test-slave-ctw4"
I20250905 02:22:52.284585 21212 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250905 02:22:52.289875 21228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:52.290828 21212 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250905 02:22:52.291153 21212 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9efda4364798446e8af7e1b1fe6fabc8"
format_stamp: "Formatted at 2025-09-05 02:22:52 on dist-test-slave-ctw4"
I20250905 02:22:52.291474 21212 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:52.343230 21212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:52.344779 21212 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:52.345242 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:52.418818 21212 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:39469
I20250905 02:22:52.418895 21280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:39469 every 8 connection(s)
I20250905 02:22:52.421494 21212 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 02:22:52.426573 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 21212
I20250905 02:22:52.427299 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 02:22:52.428133 21281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:52.452883 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8: Bootstrap starting.
I20250905 02:22:52.459177 21281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:52.461398 21281 log.cc:826] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:52.465713 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8: No bootstrap required, opened a new log
I20250905 02:22:52.484018 21281 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efda4364798446e8af7e1b1fe6fabc8" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39469 } }
I20250905 02:22:52.484678 21281 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:52.484949 21281 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efda4364798446e8af7e1b1fe6fabc8, State: Initialized, Role: FOLLOWER
I20250905 02:22:52.485648 21281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efda4364798446e8af7e1b1fe6fabc8" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39469 } }
I20250905 02:22:52.486156 21281 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:52.486429 21281 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:52.486732 21281 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:52.491102 21281 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efda4364798446e8af7e1b1fe6fabc8" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39469 } }
I20250905 02:22:52.491811 21281 leader_election.cc:304] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9efda4364798446e8af7e1b1fe6fabc8; no voters:
I20250905 02:22:52.493413 21281 leader_election.cc:290] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:22:52.494175 21286 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:52.496649 21286 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [term 1 LEADER]: Becoming Leader. State: Replica: 9efda4364798446e8af7e1b1fe6fabc8, State: Running, Role: LEADER
I20250905 02:22:52.497534 21281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:22:52.497430 21286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efda4364798446e8af7e1b1fe6fabc8" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39469 } }
I20250905 02:22:52.508527 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9efda4364798446e8af7e1b1fe6fabc8. Latest consensus state: current_term: 1 leader_uuid: "9efda4364798446e8af7e1b1fe6fabc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efda4364798446e8af7e1b1fe6fabc8" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39469 } } }
I20250905 02:22:52.509330 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:52.507054 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9efda4364798446e8af7e1b1fe6fabc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efda4364798446e8af7e1b1fe6fabc8" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 39469 } } }
I20250905 02:22:52.510717 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8 [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:52.512380 21295 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:22:52.526455 21295 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:22:52.543133 21295 catalog_manager.cc:1349] Generated new cluster ID: 8467634747214f6f960506ebb74b9641
I20250905 02:22:52.543468 21295 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:22:52.573895 21295 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:22:52.575435 21295 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:22:52.587966 21295 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9efda4364798446e8af7e1b1fe6fabc8: Generated new TSK 0
I20250905 02:22:52.588915 21295 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:22:52.598505 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:0
--local_ip_for_outbound_sockets=127.15.105.1
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39469
--builtin_ntp_servers=127.15.105.20:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 02:22:52.913864 21305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:52.914436 21305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:52.914963 21305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:52.947103 21305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:52.948012 21305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:22:52.985983 21305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:39469
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:52.987331 21305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:52.989259 21305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:53.001940 21311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:54.405903 21310 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 21305
W20250905 02:22:54.949918 21305 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.947s user 0.649s sys 1.232s
W20250905 02:22:54.950766 21305 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.948s user 0.649s sys 1.232s
W20250905 02:22:53.003142 21312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:54.952076 21313 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1947 milliseconds
W20250905 02:22:54.953392 21314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:54.953327 21305 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:54.956614 21305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:54.958896 21305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:54.960320 21305 hybrid_clock.cc:648] HybridClock initialized: now 1757038974960289 us; error 44 us; skew 500 ppm
I20250905 02:22:54.961227 21305 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:54.968364 21305 webserver.cc:480] Webserver started at http://127.15.105.1:36081/ using document root <none> and password file <none>
I20250905 02:22:54.969409 21305 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:54.969653 21305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:54.970145 21305 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:54.976320 21305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "99311c5728984f619e51d55cf7ca4004"
format_stamp: "Formatted at 2025-09-05 02:22:54 on dist-test-slave-ctw4"
I20250905 02:22:54.977710 21305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "99311c5728984f619e51d55cf7ca4004"
format_stamp: "Formatted at 2025-09-05 02:22:54 on dist-test-slave-ctw4"
I20250905 02:22:54.985904 21305 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.005s
I20250905 02:22:54.992221 21321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:54.993561 21305 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250905 02:22:54.993939 21305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "99311c5728984f619e51d55cf7ca4004"
format_stamp: "Formatted at 2025-09-05 02:22:54 on dist-test-slave-ctw4"
I20250905 02:22:54.994308 21305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:55.044862 21305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:55.046422 21305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:22:55.046916 21305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:55.050108 21305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:55.054558 21305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:55.054802 21305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:55.055094 21305 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:55.055259 21305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:55.231914 21305 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:43189
I20250905 02:22:55.232138 21433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:43189 every 8 connection(s)
I20250905 02:22:55.234633 21305 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 02:22:55.244865 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 21305
I20250905 02:22:55.245420 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757038821926622-15780-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 02:22:55.261392 21434 heartbeater.cc:344] Connected to a master server at 127.15.105.62:39469
I20250905 02:22:55.261894 21434 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:55.263204 21434 heartbeater.cc:507] Master 127.15.105.62:39469 requested a full tablet report, sending...
I20250905 02:22:55.266361 21246 ts_manager.cc:194] Registered new tserver with Master: 99311c5728984f619e51d55cf7ca4004 (127.15.105.1:43189)
I20250905 02:22:55.269358 21246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:49631
I20250905 02:22:55.281630 15780 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250905 02:22:55.321476 21246 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55856:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250905 02:22:55.402822 21369 tablet_service.cc:1468] Processing CreateTablet for tablet 97a59593107f47fe90b93b8a9160901e (DEFAULT_TABLE table=TestTable [id=ef957bbe7acd4d709bb1022b75c78a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:22:55.404472 21369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97a59593107f47fe90b93b8a9160901e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:55.432008 21449 tablet_bootstrap.cc:492] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004: Bootstrap starting.
I20250905 02:22:55.439741 21449 tablet_bootstrap.cc:654] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:55.442263 21449 log.cc:826] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004: Log is configured to *not* fsync() on all Append() calls
I20250905 02:22:55.448343 21449 tablet_bootstrap.cc:492] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004: No bootstrap required, opened a new log
I20250905 02:22:55.448890 21449 ts_tablet_manager.cc:1397] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20250905 02:22:55.477272 21449 raft_consensus.cc:357] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99311c5728984f619e51d55cf7ca4004" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 43189 } }
I20250905 02:22:55.478022 21449 raft_consensus.cc:383] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:55.478322 21449 raft_consensus.cc:738] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99311c5728984f619e51d55cf7ca4004, State: Initialized, Role: FOLLOWER
I20250905 02:22:55.479266 21449 consensus_queue.cc:260] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99311c5728984f619e51d55cf7ca4004" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 43189 } }
I20250905 02:22:55.480165 21449 raft_consensus.cc:397] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:55.480569 21449 raft_consensus.cc:491] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:55.481002 21449 raft_consensus.cc:3058] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:55.487598 21449 raft_consensus.cc:513] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99311c5728984f619e51d55cf7ca4004" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 43189 } }
I20250905 02:22:55.488588 21449 leader_election.cc:304] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99311c5728984f619e51d55cf7ca4004; no voters:
I20250905 02:22:55.490936 21449 leader_election.cc:290] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:22:55.491227 21451 raft_consensus.cc:2802] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:55.493968 21434 heartbeater.cc:499] Master 127.15.105.62:39469 was elected leader, sending a full tablet report...
I20250905 02:22:55.495303 21449 ts_tablet_manager.cc:1428] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004: Time spent starting tablet: real 0.046s user 0.039s sys 0.006s
I20250905 02:22:55.497153 21451 raft_consensus.cc:695] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [term 1 LEADER]: Becoming Leader. State: Replica: 99311c5728984f619e51d55cf7ca4004, State: Running, Role: LEADER
I20250905 02:22:55.498482 21451 consensus_queue.cc:237] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99311c5728984f619e51d55cf7ca4004" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 43189 } }
I20250905 02:22:55.510077 21246 catalog_manager.cc:5582] T 97a59593107f47fe90b93b8a9160901e P 99311c5728984f619e51d55cf7ca4004 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99311c5728984f619e51d55cf7ca4004 (127.15.105.1). New cstate: current_term: 1 leader_uuid: "99311c5728984f619e51d55cf7ca4004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99311c5728984f619e51d55cf7ca4004" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 43189 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:55.530481 15780 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250905 02:22:55.533666 15780 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 99311c5728984f619e51d55cf7ca4004 to finish bootstrapping
I20250905 02:22:58.239382 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 21305
I20250905 02:22:58.268566 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 21212
2025-09-05T02:22:58Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (8311 ms)
[----------] 1 test from ListTableCliSimpleParamTest (8311 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250905 02:22:58.326400 15780 test_util.cc:276] Using random seed: -2077696153
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (13 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-09-05T02:22:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T02:22:58Z Disabled control of system clock
I20250905 02:22:58.380903 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:34739
--webserver_interface=127.15.105.62
--webserver_port=0
--builtin_ntp_servers=127.15.105.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:34739 with env {}
W20250905 02:22:58.687534 21477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:22:58.688127 21477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:22:58.688544 21477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:22:58.720532 21477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:22:58.720846 21477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:22:58.721055 21477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:22:58.721248 21477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:22:58.757448 21477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:34739
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:22:58.758709 21477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:22:58.760409 21477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:58.771129 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:58.773547 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:59.978052 21485 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1200 milliseconds
W20250905 02:22:59.979156 21477 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.206s user 0.379s sys 0.803s
W20250905 02:22:59.979497 21477 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.207s user 0.379s sys 0.803s
W20250905 02:22:59.979521 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:59.979769 21477 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:22:59.981022 21477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:22:59.990244 21477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:22:59.991696 21477 hybrid_clock.cc:648] HybridClock initialized: now 1757038979991661 us; error 51 us; skew 500 ppm
I20250905 02:22:59.992702 21477 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:00.001392 21477 webserver.cc:480] Webserver started at http://127.15.105.62:43511/ using document root <none> and password file <none>
I20250905 02:23:00.002408 21477 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:00.002630 21477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:00.003104 21477 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:23:00.007936 21477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/instance:
uuid: "0c6a63bb35ca4964b61aca2f0324320b"
format_stamp: "Formatted at 2025-09-05 02:22:59 on dist-test-slave-ctw4"
I20250905 02:23:00.009308 21477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal/instance:
uuid: "0c6a63bb35ca4964b61aca2f0324320b"
format_stamp: "Formatted at 2025-09-05 02:22:59 on dist-test-slave-ctw4"
I20250905 02:23:00.020218 21477 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.009s sys 0.002s
I20250905 02:23:00.029160 21493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:00.030635 21477 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250905 02:23:00.031131 21477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
uuid: "0c6a63bb35ca4964b61aca2f0324320b"
format_stamp: "Formatted at 2025-09-05 02:22:59 on dist-test-slave-ctw4"
I20250905 02:23:00.031611 21477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:00.116235 21477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:00.117781 21477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:00.118289 21477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:00.192098 21477 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:34739
I20250905 02:23:00.192178 21544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:34739 every 8 connection(s)
I20250905 02:23:00.194891 21477 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/info.pb
I20250905 02:23:00.196835 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 21477
I20250905 02:23:00.197465 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal/instance
I20250905 02:23:00.202816 21545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.230118 21545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b: Bootstrap starting.
I20250905 02:23:00.236404 21545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.238229 21545 log.cc:826] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b: Log is configured to *not* fsync() on all Append() calls
I20250905 02:23:00.242900 21545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b: No bootstrap required, opened a new log
I20250905 02:23:00.261557 21545 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6a63bb35ca4964b61aca2f0324320b" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } }
I20250905 02:23:00.262286 21545 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.262490 21545 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6a63bb35ca4964b61aca2f0324320b, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.263224 21545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6a63bb35ca4964b61aca2f0324320b" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } }
I20250905 02:23:00.263870 21545 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:23:00.264228 21545 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:23:00.264602 21545 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.269243 21545 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6a63bb35ca4964b61aca2f0324320b" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } }
I20250905 02:23:00.269992 21545 leader_election.cc:304] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c6a63bb35ca4964b61aca2f0324320b; no voters:
I20250905 02:23:00.271649 21545 leader_election.cc:290] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:23:00.272549 21550 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:00.275213 21550 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [term 1 LEADER]: Becoming Leader. State: Replica: 0c6a63bb35ca4964b61aca2f0324320b, State: Running, Role: LEADER
I20250905 02:23:00.276407 21545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:23:00.276219 21550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6a63bb35ca4964b61aca2f0324320b" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } }
I20250905 02:23:00.285777 21551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c6a63bb35ca4964b61aca2f0324320b. Latest consensus state: current_term: 1 leader_uuid: "0c6a63bb35ca4964b61aca2f0324320b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6a63bb35ca4964b61aca2f0324320b" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } } }
I20250905 02:23:00.286523 21551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [sys.catalog]: This master's current role is: LEADER
I20250905 02:23:00.287513 21552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c6a63bb35ca4964b61aca2f0324320b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c6a63bb35ca4964b61aca2f0324320b" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } } }
I20250905 02:23:00.288451 21552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b [sys.catalog]: This master's current role is: LEADER
I20250905 02:23:00.291129 21559 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:23:00.304716 21559 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:23:00.324132 21559 catalog_manager.cc:1349] Generated new cluster ID: 368ab7547ed540309f4866bc2a767720
I20250905 02:23:00.324486 21559 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:23:00.349931 21559 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:23:00.351497 21559 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:23:00.364833 21559 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0c6a63bb35ca4964b61aca2f0324320b: Generated new TSK 0
I20250905 02:23:00.365856 21559 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:23:00.380949 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:0
--local_ip_for_outbound_sockets=127.15.105.1
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:34739
--builtin_ntp_servers=127.15.105.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250905 02:23:00.712845 21569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:23:00.713367 21569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:23:00.713878 21569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:23:00.745625 21569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:23:00.746501 21569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:23:00.782033 21569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=0
--tserver_master_addrs=127.15.105.62:34739
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:23:00.783309 21569 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:23:00.784909 21569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:23:00.797611 21575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:02.201553 21574 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 21569
W20250905 02:23:02.244861 21574 kernel_stack_watchdog.cc:198] Thread 21569 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:23:02.245177 21577 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250905 02:23:00.798786 21576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:02.251021 21569 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.452s user 0.579s sys 0.872s
W20250905 02:23:02.251387 21569 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.453s user 0.579s sys 0.872s
W20250905 02:23:02.253019 21578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:02.253033 21569 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:23:02.254292 21569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:23:02.257081 21569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:23:02.258600 21569 hybrid_clock.cc:648] HybridClock initialized: now 1757038982258530 us; error 90 us; skew 500 ppm
I20250905 02:23:02.259454 21569 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:02.266619 21569 webserver.cc:480] Webserver started at http://127.15.105.1:40945/ using document root <none> and password file <none>
I20250905 02:23:02.267644 21569 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:02.267933 21569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:02.268433 21569 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:23:02.273175 21569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/instance:
uuid: "5d8edf8fb0c74f6fa51f001510404cdc"
format_stamp: "Formatted at 2025-09-05 02:23:02 on dist-test-slave-ctw4"
I20250905 02:23:02.274389 21569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal/instance:
uuid: "5d8edf8fb0c74f6fa51f001510404cdc"
format_stamp: "Formatted at 2025-09-05 02:23:02 on dist-test-slave-ctw4"
I20250905 02:23:02.282845 21569 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20250905 02:23:02.289651 21585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:02.291117 21569 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.005s
I20250905 02:23:02.291586 21569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
uuid: "5d8edf8fb0c74f6fa51f001510404cdc"
format_stamp: "Formatted at 2025-09-05 02:23:02 on dist-test-slave-ctw4"
I20250905 02:23:02.292111 21569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:02.376178 21569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:02.377791 21569 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:02.378412 21569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:02.381635 21569 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:23:02.386770 21569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:23:02.387027 21569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:02.387291 21569 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:23:02.387454 21569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:02.570870 21569 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:40643
I20250905 02:23:02.570988 21698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:40643 every 8 connection(s)
I20250905 02:23:02.573705 21569 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/info.pb
I20250905 02:23:02.574506 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 21569
I20250905 02:23:02.575275 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal/instance
I20250905 02:23:02.584627 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:0
--local_ip_for_outbound_sockets=127.15.105.2
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:34739
--builtin_ntp_servers=127.15.105.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 02:23:02.617125 21699 heartbeater.cc:344] Connected to a master server at 127.15.105.62:34739
I20250905 02:23:02.617605 21699 heartbeater.cc:461] Registering TS with master...
I20250905 02:23:02.618741 21699 heartbeater.cc:507] Master 127.15.105.62:34739 requested a full tablet report, sending...
I20250905 02:23:02.621589 21510 ts_manager.cc:194] Registered new tserver with Master: 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1:40643)
I20250905 02:23:02.623656 21510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:46651
W20250905 02:23:02.921427 21703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:23:02.921973 21703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:23:02.922501 21703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:23:02.954814 21703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:23:02.955734 21703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:23:02.991245 21703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=0
--tserver_master_addrs=127.15.105.62:34739
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:23:02.992666 21703 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:23:02.994333 21703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:23:03.006189 21709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:03.627803 21699 heartbeater.cc:499] Master 127.15.105.62:34739 was elected leader, sending a full tablet report...
W20250905 02:23:03.007369 21710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:04.191169 21711 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250905 02:23:04.192540 21712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:04.192588 21703 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:23:04.196431 21703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:23:04.199263 21703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:23:04.200817 21703 hybrid_clock.cc:648] HybridClock initialized: now 1757038984200769 us; error 61 us; skew 500 ppm
I20250905 02:23:04.201676 21703 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:04.209111 21703 webserver.cc:480] Webserver started at http://127.15.105.2:43355/ using document root <none> and password file <none>
I20250905 02:23:04.210134 21703 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:04.210368 21703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:04.210836 21703 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:23:04.215945 21703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/instance:
uuid: "f3b55e4b3af944a18c843b4bdc74a7a5"
format_stamp: "Formatted at 2025-09-05 02:23:04 on dist-test-slave-ctw4"
I20250905 02:23:04.217172 21703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal/instance:
uuid: "f3b55e4b3af944a18c843b4bdc74a7a5"
format_stamp: "Formatted at 2025-09-05 02:23:04 on dist-test-slave-ctw4"
I20250905 02:23:04.224805 21703 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250905 02:23:04.230953 21719 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:04.232157 21703 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250905 02:23:04.232529 21703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
uuid: "f3b55e4b3af944a18c843b4bdc74a7a5"
format_stamp: "Formatted at 2025-09-05 02:23:04 on dist-test-slave-ctw4"
I20250905 02:23:04.232898 21703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:04.292718 21703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:04.294270 21703 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:04.294744 21703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:04.297338 21703 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:23:04.301738 21703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:23:04.301990 21703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:04.302258 21703 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:23:04.302428 21703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:04.444258 21703 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:44381
I20250905 02:23:04.444365 21831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:44381 every 8 connection(s)
I20250905 02:23:04.446908 21703 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/info.pb
I20250905 02:23:04.451855 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 21703
I20250905 02:23:04.452435 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal/instance
I20250905 02:23:04.459100 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:0
--local_ip_for_outbound_sockets=127.15.105.3
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:34739
--builtin_ntp_servers=127.15.105.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 02:23:04.470301 21832 heartbeater.cc:344] Connected to a master server at 127.15.105.62:34739
I20250905 02:23:04.470885 21832 heartbeater.cc:461] Registering TS with master...
I20250905 02:23:04.472385 21832 heartbeater.cc:507] Master 127.15.105.62:34739 requested a full tablet report, sending...
I20250905 02:23:04.474959 21510 ts_manager.cc:194] Registered new tserver with Master: f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381)
I20250905 02:23:04.476300 21510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:37617
W20250905 02:23:04.762637 21836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:23:04.763103 21836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:23:04.763617 21836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:23:04.794883 21836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:23:04.795819 21836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:23:04.829241 21836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=0
--tserver_master_addrs=127.15.105.62:34739
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:23:04.830582 21836 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:23:04.832084 21836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:23:04.842350 21842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:05.480756 21832 heartbeater.cc:499] Master 127.15.105.62:34739 was elected leader, sending a full tablet report...
W20250905 02:23:04.843937 21843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:06.039326 21845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:06.041941 21844 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1194 milliseconds
I20250905 02:23:06.042028 21836 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:23:06.043172 21836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:23:06.046481 21836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:23:06.047883 21836 hybrid_clock.cc:648] HybridClock initialized: now 1757038986047847 us; error 52 us; skew 500 ppm
I20250905 02:23:06.048725 21836 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:06.055358 21836 webserver.cc:480] Webserver started at http://127.15.105.3:36243/ using document root <none> and password file <none>
I20250905 02:23:06.056289 21836 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:06.056469 21836 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:06.056881 21836 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:23:06.061177 21836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/instance:
uuid: "b76ad76b8be24a269131e185d694499c"
format_stamp: "Formatted at 2025-09-05 02:23:06 on dist-test-slave-ctw4"
I20250905 02:23:06.062240 21836 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal/instance:
uuid: "b76ad76b8be24a269131e185d694499c"
format_stamp: "Formatted at 2025-09-05 02:23:06 on dist-test-slave-ctw4"
I20250905 02:23:06.069329 21836 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250905 02:23:06.074813 21852 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:06.075901 21836 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 02:23:06.076251 21836 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
uuid: "b76ad76b8be24a269131e185d694499c"
format_stamp: "Formatted at 2025-09-05 02:23:06 on dist-test-slave-ctw4"
I20250905 02:23:06.076589 21836 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:06.135278 21836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:06.136819 21836 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:06.137262 21836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:06.139798 21836 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:23:06.143987 21836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:23:06.144209 21836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:06.144452 21836 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:23:06.144610 21836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:06.281284 21836 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:42637
I20250905 02:23:06.281379 21964 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:42637 every 8 connection(s)
I20250905 02:23:06.283754 21836 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/info.pb
I20250905 02:23:06.290045 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 21836
I20250905 02:23:06.290438 15780 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal/instance
I20250905 02:23:06.307549 21965 heartbeater.cc:344] Connected to a master server at 127.15.105.62:34739
I20250905 02:23:06.308051 21965 heartbeater.cc:461] Registering TS with master...
I20250905 02:23:06.309185 21965 heartbeater.cc:507] Master 127.15.105.62:34739 requested a full tablet report, sending...
I20250905 02:23:06.311347 21510 ts_manager.cc:194] Registered new tserver with Master: b76ad76b8be24a269131e185d694499c (127.15.105.3:42637)
I20250905 02:23:06.312608 21510 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:60885
I20250905 02:23:06.324115 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:23:06.353452 15780 test_util.cc:276] Using random seed: -2069669095
I20250905 02:23:06.392511 21510 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40218:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:23:06.394872 21510 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:23:06.450112 21767 tablet_service.cc:1468] Processing CreateTablet for tablet bfa1d90e9c09436490ac65d5498ad2ca (DEFAULT_TABLE table=pre_rebuild [id=d01dab0a0e524516b26ed4738e0e296a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:23:06.451267 21634 tablet_service.cc:1468] Processing CreateTablet for tablet bfa1d90e9c09436490ac65d5498ad2ca (DEFAULT_TABLE table=pre_rebuild [id=d01dab0a0e524516b26ed4738e0e296a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:23:06.452410 21767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa1d90e9c09436490ac65d5498ad2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:06.453290 21634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa1d90e9c09436490ac65d5498ad2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:06.453573 21900 tablet_service.cc:1468] Processing CreateTablet for tablet bfa1d90e9c09436490ac65d5498ad2ca (DEFAULT_TABLE table=pre_rebuild [id=d01dab0a0e524516b26ed4738e0e296a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:23:06.458855 21900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa1d90e9c09436490ac65d5498ad2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:06.481818 21989 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Bootstrap starting.
I20250905 02:23:06.483742 21990 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Bootstrap starting.
I20250905 02:23:06.489831 21989 tablet_bootstrap.cc:654] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:06.492480 21989 log.cc:826] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Log is configured to *not* fsync() on all Append() calls
I20250905 02:23:06.494711 21990 tablet_bootstrap.cc:654] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:06.496917 21991 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Bootstrap starting.
I20250905 02:23:06.497243 21990 log.cc:826] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Log is configured to *not* fsync() on all Append() calls
I20250905 02:23:06.498150 21989 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: No bootstrap required, opened a new log
I20250905 02:23:06.498575 21989 ts_tablet_manager.cc:1397] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20250905 02:23:06.503593 21990 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: No bootstrap required, opened a new log
I20250905 02:23:06.504254 21990 ts_tablet_manager.cc:1397] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20250905 02:23:06.504736 21991 tablet_bootstrap.cc:654] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:06.507171 21991 log.cc:826] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Log is configured to *not* fsync() on all Append() calls
I20250905 02:23:06.518132 21989 raft_consensus.cc:357] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.518792 21989 raft_consensus.cc:383] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:06.519061 21989 raft_consensus.cc:738] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b55e4b3af944a18c843b4bdc74a7a5, State: Initialized, Role: FOLLOWER
I20250905 02:23:06.519634 21991 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: No bootstrap required, opened a new log
I20250905 02:23:06.520201 21991 ts_tablet_manager.cc:1397] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.016s
I20250905 02:23:06.519862 21989 consensus_queue.cc:260] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.525776 21989 ts_tablet_manager.cc:1428] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Time spent starting tablet: real 0.027s user 0.019s sys 0.006s
I20250905 02:23:06.532912 21990 raft_consensus.cc:357] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.533860 21990 raft_consensus.cc:383] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:06.534263 21990 raft_consensus.cc:738] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8edf8fb0c74f6fa51f001510404cdc, State: Initialized, Role: FOLLOWER
I20250905 02:23:06.535390 21990 consensus_queue.cc:260] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.539505 21990 ts_tablet_manager.cc:1428] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Time spent starting tablet: real 0.035s user 0.032s sys 0.001s
I20250905 02:23:06.545490 21991 raft_consensus.cc:357] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.546160 21991 raft_consensus.cc:383] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:06.546386 21991 raft_consensus.cc:738] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76ad76b8be24a269131e185d694499c, State: Initialized, Role: FOLLOWER
I20250905 02:23:06.547155 21991 consensus_queue.cc:260] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.549602 21965 heartbeater.cc:499] Master 127.15.105.62:34739 was elected leader, sending a full tablet report...
I20250905 02:23:06.550992 21991 ts_tablet_manager.cc:1428] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Time spent starting tablet: real 0.030s user 0.024s sys 0.004s
W20250905 02:23:06.601415 21700 tablet.cc:2378] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:23:06.628552 21995 raft_consensus.cc:491] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:06.629011 21995 raft_consensus.cc:513] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.631306 21995 leader_election.cc:290] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b76ad76b8be24a269131e185d694499c (127.15.105.3:42637), 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1:40643)
I20250905 02:23:06.642654 21920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76ad76b8be24a269131e185d694499c" is_pre_election: true
I20250905 02:23:06.643388 21920 raft_consensus.cc:2466] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b55e4b3af944a18c843b4bdc74a7a5 in term 0.
I20250905 02:23:06.644711 21720 leader_election.cc:304] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b76ad76b8be24a269131e185d694499c, f3b55e4b3af944a18c843b4bdc74a7a5; no voters:
I20250905 02:23:06.644671 21654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" is_pre_election: true
I20250905 02:23:06.645596 21995 raft_consensus.cc:2802] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:06.645563 21654 raft_consensus.cc:2466] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3b55e4b3af944a18c843b4bdc74a7a5 in term 0.
I20250905 02:23:06.645892 21995 raft_consensus.cc:491] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:06.646150 21995 raft_consensus.cc:3058] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:06.650501 21995 raft_consensus.cc:513] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.651968 21995 leader_election.cc:290] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [CANDIDATE]: Term 1 election: Requested vote from peers b76ad76b8be24a269131e185d694499c (127.15.105.3:42637), 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1:40643)
I20250905 02:23:06.652770 21920 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76ad76b8be24a269131e185d694499c"
I20250905 02:23:06.653208 21920 raft_consensus.cc:3058] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:06.653522 21654 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d8edf8fb0c74f6fa51f001510404cdc"
I20250905 02:23:06.653949 21654 raft_consensus.cc:3058] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:06.657833 21920 raft_consensus.cc:2466] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b55e4b3af944a18c843b4bdc74a7a5 in term 1.
I20250905 02:23:06.658413 21654 raft_consensus.cc:2466] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3b55e4b3af944a18c843b4bdc74a7a5 in term 1.
I20250905 02:23:06.658818 21720 leader_election.cc:304] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b76ad76b8be24a269131e185d694499c, f3b55e4b3af944a18c843b4bdc74a7a5; no voters:
I20250905 02:23:06.659468 21995 raft_consensus.cc:2802] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:06.661033 21995 raft_consensus.cc:695] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 LEADER]: Becoming Leader. State: Replica: f3b55e4b3af944a18c843b4bdc74a7a5, State: Running, Role: LEADER
I20250905 02:23:06.661912 21995 consensus_queue.cc:237] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:06.673089 21510 catalog_manager.cc:5582] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2). New cstate: current_term: 1 leader_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } health_report { overall_health: UNKNOWN } } }
W20250905 02:23:06.709693 21833 tablet.cc:2378] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 02:23:06.788558 21966 tablet.cc:2378] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:23:06.824851 21654 raft_consensus.cc:1273] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Refusing update from remote peer f3b55e4b3af944a18c843b4bdc74a7a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:23:06.825063 21920 raft_consensus.cc:1273] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 1 FOLLOWER]: Refusing update from remote peer f3b55e4b3af944a18c843b4bdc74a7a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:23:06.826737 22000 consensus_queue.cc:1035] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:23:06.827543 21995 consensus_queue.cc:1035] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:23:06.864116 22007 mvcc.cc:204] Tried to move back new op lower bound from 7196831690016813056 to 7196831689374371840. Current Snapshot: MvccSnapshot[applied={T|T < 7196831690016813056}]
I20250905 02:23:06.872356 22009 mvcc.cc:204] Tried to move back new op lower bound from 7196831690016813056 to 7196831689374371840. Current Snapshot: MvccSnapshot[applied={T|T < 7196831690016813056}]
I20250905 02:23:12.225690 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 21477
W20250905 02:23:12.647310 22046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:23:12.648211 22046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:23:12.687047 22046 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250905 02:23:13.006711 21965 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:34739 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:34739: connect: Connection refused (error 111)
W20250905 02:23:13.010404 21699 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:34739 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:34739: connect: Connection refused (error 111)
W20250905 02:23:13.014878 21832 heartbeater.cc:646] Failed to heartbeat to 127.15.105.62:34739 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.105.62:34739: connect: Connection refused (error 111)
W20250905 02:23:14.136283 22054 debug-util.cc:398] Leaking SignalData structure 0x7b08000373e0 after lost signal to thread 22046
W20250905 02:23:14.136888 22054 kernel_stack_watchdog.cc:198] Thread 22046 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:23:14.241475 22046 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.509s user 0.557s sys 0.947s
W20250905 02:23:14.361721 22046 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.629s user 0.558s sys 0.959s
I20250905 02:23:14.439121 22046 minidump.cc:252] Setting minidump size limit to 20M
I20250905 02:23:14.441155 22046 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:23:14.442279 22046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:23:14.452478 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:14.453944 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:14.456597 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:14.465251 22046 server_base.cc:1047] running on GCE node
I20250905 02:23:14.466468 22046 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250905 02:23:14.466971 22046 hybrid_clock.cc:648] HybridClock initialized: now 1757038994466942 us; error 38115 us; skew 500 ppm
I20250905 02:23:14.467839 22046 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:14.472736 22046 webserver.cc:480] Webserver started at http://0.0.0.0:38277/ using document root <none> and password file <none>
I20250905 02:23:14.473760 22046 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:14.474043 22046 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:14.474575 22046 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:23:14.479367 22046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/instance:
uuid: "fac50144742b4920a90057bedebab6e1"
format_stamp: "Formatted at 2025-09-05 02:23:14 on dist-test-slave-ctw4"
I20250905 02:23:14.480392 22046 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal/instance:
uuid: "fac50144742b4920a90057bedebab6e1"
format_stamp: "Formatted at 2025-09-05 02:23:14 on dist-test-slave-ctw4"
I20250905 02:23:14.486419 22046 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250905 02:23:14.490842 22090 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:14.491674 22046 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 02:23:14.492132 22046 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
uuid: "fac50144742b4920a90057bedebab6e1"
format_stamp: "Formatted at 2025-09-05 02:23:14 on dist-test-slave-ctw4"
I20250905 02:23:14.492435 22046 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:14.618743 22046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:14.620213 22046 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:14.620604 22046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:14.624913 22046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:14.639338 22046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Bootstrap starting.
I20250905 02:23:14.644300 22046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:14.645892 22046 log.cc:826] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Log is configured to *not* fsync() on all Append() calls
I20250905 02:23:14.649999 22046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: No bootstrap required, opened a new log
I20250905 02:23:14.666491 22046 raft_consensus.cc:357] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER }
I20250905 02:23:14.667011 22046 raft_consensus.cc:383] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:14.667191 22046 raft_consensus.cc:738] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac50144742b4920a90057bedebab6e1, State: Initialized, Role: FOLLOWER
I20250905 02:23:14.667899 22046 consensus_queue.cc:260] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER }
I20250905 02:23:14.668365 22046 raft_consensus.cc:397] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:23:14.668632 22046 raft_consensus.cc:491] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:23:14.669013 22046 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:14.673501 22046 raft_consensus.cc:513] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER }
I20250905 02:23:14.674216 22046 leader_election.cc:304] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fac50144742b4920a90057bedebab6e1; no voters:
I20250905 02:23:14.675921 22046 leader_election.cc:290] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:23:14.676165 22097 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:14.678867 22097 raft_consensus.cc:695] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 LEADER]: Becoming Leader. State: Replica: fac50144742b4920a90057bedebab6e1, State: Running, Role: LEADER
I20250905 02:23:14.679769 22097 consensus_queue.cc:237] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER }
I20250905 02:23:14.687383 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fac50144742b4920a90057bedebab6e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER } }
I20250905 02:23:14.687330 22099 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fac50144742b4920a90057bedebab6e1. Latest consensus state: current_term: 1 leader_uuid: "fac50144742b4920a90057bedebab6e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER } }
I20250905 02:23:14.688035 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: This master's current role is: LEADER
I20250905 02:23:14.688047 22099 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: This master's current role is: LEADER
I20250905 02:23:14.701265 22046 tablet_replica.cc:331] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: stopping tablet replica
I20250905 02:23:14.701815 22046 raft_consensus.cc:2241] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:14.702258 22046 raft_consensus.cc:2270] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:14.704429 22046 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250905 02:23:14.704871 22046 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250905 02:23:14.760190 22046 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250905 02:23:15.794735 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 21569
I20250905 02:23:15.829450 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 21703
I20250905 02:23:15.868134 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 21836
I20250905 02:23:15.902100 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:34739
--webserver_interface=127.15.105.62
--webserver_port=43511
--builtin_ntp_servers=127.15.105.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.105.62:34739 with env {}
W20250905 02:23:16.204268 22108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:23:16.204852 22108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:23:16.205309 22108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:23:16.236159 22108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 02:23:16.236491 22108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:23:16.236711 22108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 02:23:16.236912 22108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 02:23:16.271121 22108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.105.62:34739
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.105.62:34739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.105.62
--webserver_port=43511
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:23:16.272341 22108 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:23:16.273800 22108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:23:16.284933 22114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:17.688520 22113 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 22108
W20250905 02:23:18.011847 22113 kernel_stack_watchdog.cc:198] Thread 22108 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:23:18.012470 22108 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.728s user 0.615s sys 1.106s
W20250905 02:23:18.012871 22108 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.728s user 0.615s sys 1.106s
W20250905 02:23:16.285553 22115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:18.014987 22117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:18.018509 22116 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1727 milliseconds
I20250905 02:23:18.018560 22108 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:23:18.020126 22108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:23:18.022953 22108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:23:18.024276 22108 hybrid_clock.cc:648] HybridClock initialized: now 1757038998024235 us; error 42 us; skew 500 ppm
I20250905 02:23:18.025012 22108 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:18.030807 22108 webserver.cc:480] Webserver started at http://127.15.105.62:43511/ using document root <none> and password file <none>
I20250905 02:23:18.031656 22108 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:18.031931 22108 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:18.039526 22108 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.007s
I20250905 02:23:18.043973 22124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:18.044924 22108 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250905 02:23:18.045240 22108 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
uuid: "fac50144742b4920a90057bedebab6e1"
format_stamp: "Formatted at 2025-09-05 02:23:14 on dist-test-slave-ctw4"
I20250905 02:23:18.047070 22108 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:18.091403 22108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:18.092841 22108 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:18.093250 22108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:18.162111 22108 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.62:34739
I20250905 02:23:18.162199 22175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.62:34739 every 8 connection(s)
I20250905 02:23:18.164925 22108 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/info.pb
I20250905 02:23:18.173604 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 22108
I20250905 02:23:18.175068 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.1:40643
--local_ip_for_outbound_sockets=127.15.105.1
--tserver_master_addrs=127.15.105.62:34739
--webserver_port=40945
--webserver_interface=127.15.105.1
--builtin_ntp_servers=127.15.105.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 02:23:18.175192 22176 sys_catalog.cc:263] Verifying existing consensus state
I20250905 02:23:18.190682 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Bootstrap starting.
I20250905 02:23:18.206064 22176 log.cc:826] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Log is configured to *not* fsync() on all Append() calls
I20250905 02:23:18.220768 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:18.221752 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Bootstrap complete.
I20250905 02:23:18.251358 22176 raft_consensus.cc:357] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } }
I20250905 02:23:18.252101 22176 raft_consensus.cc:738] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac50144742b4920a90057bedebab6e1, State: Initialized, Role: FOLLOWER
I20250905 02:23:18.252890 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } }
I20250905 02:23:18.253392 22176 raft_consensus.cc:397] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:23:18.253655 22176 raft_consensus.cc:491] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:23:18.253954 22176 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:18.258044 22176 raft_consensus.cc:513] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } }
I20250905 02:23:18.258736 22176 leader_election.cc:304] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fac50144742b4920a90057bedebab6e1; no voters:
I20250905 02:23:18.260895 22176 leader_election.cc:290] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 02:23:18.261296 22180 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:23:18.264418 22180 raft_consensus.cc:695] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [term 2 LEADER]: Becoming Leader. State: Replica: fac50144742b4920a90057bedebab6e1, State: Running, Role: LEADER
I20250905 02:23:18.265244 22180 consensus_queue.cc:237] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } }
I20250905 02:23:18.266088 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:23:18.275019 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fac50144742b4920a90057bedebab6e1. Latest consensus state: current_term: 2 leader_uuid: "fac50144742b4920a90057bedebab6e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } } }
I20250905 02:23:18.275647 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: This master's current role is: LEADER
I20250905 02:23:18.275710 22181 sys_catalog.cc:455] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fac50144742b4920a90057bedebab6e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fac50144742b4920a90057bedebab6e1" member_type: VOTER last_known_addr { host: "127.15.105.62" port: 34739 } } }
I20250905 02:23:18.276401 22181 sys_catalog.cc:458] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1 [sys.catalog]: This master's current role is: LEADER
I20250905 02:23:18.282053 22188 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:23:18.295518 22188 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=d32127ef69524701b628c88782ed72b8]
I20250905 02:23:18.303349 22188 tablet_loader.cc:96] loaded metadata for tablet bfa1d90e9c09436490ac65d5498ad2ca (table pre_rebuild [id=d32127ef69524701b628c88782ed72b8])
I20250905 02:23:18.305015 22188 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:23:18.335060 22188 catalog_manager.cc:1349] Generated new cluster ID: 51e6cadbc4054859b4a21d610ab71b27
I20250905 02:23:18.335361 22188 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:23:18.361951 22188 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:23:18.363966 22188 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:23:18.376695 22199 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250905 02:23:18.390036 22188 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Generated new TSK 0
I20250905 02:23:18.391340 22188 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250905 02:23:18.563927 22178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:23:18.564424 22178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:23:18.564905 22178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:23:18.596009 22178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:23:18.596838 22178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.1
I20250905 02:23:18.630663 22178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.1:40643
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.105.1
--webserver_port=40945
--tserver_master_addrs=127.15.105.62:34739
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.1
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:23:18.631917 22178 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:23:18.633489 22178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:23:18.645212 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:20.048389 22204 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 22178
W20250905 02:23:20.121677 22204 kernel_stack_watchdog.cc:198] Thread 22178 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:23:18.646277 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:20.122633 22178 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.476s user 0.442s sys 0.912s
W20250905 02:23:20.123082 22178 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.477s user 0.442s sys 0.912s
W20250905 02:23:20.124907 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:20.128001 22207 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1481 milliseconds
I20250905 02:23:20.128006 22178 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:23:20.129467 22178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:23:20.131497 22178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:23:20.132828 22178 hybrid_clock.cc:648] HybridClock initialized: now 1757039000132782 us; error 52 us; skew 500 ppm
I20250905 02:23:20.133590 22178 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:20.139465 22178 webserver.cc:480] Webserver started at http://127.15.105.1:40945/ using document root <none> and password file <none>
I20250905 02:23:20.140377 22178 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:20.140595 22178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:20.148343 22178 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250905 02:23:20.153062 22215 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:20.154099 22178 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250905 02:23:20.154422 22178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
uuid: "5d8edf8fb0c74f6fa51f001510404cdc"
format_stamp: "Formatted at 2025-09-05 02:23:02 on dist-test-slave-ctw4"
I20250905 02:23:20.156343 22178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:20.220268 22178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:20.221685 22178 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:20.222115 22178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:20.224704 22178 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:23:20.230682 22222 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 02:23:20.242162 22178 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 02:23:20.242404 22178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250905 02:23:20.242671 22178 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 02:23:20.247020 22178 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 02:23:20.247205 22178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250905 02:23:20.247694 22222 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Bootstrap starting.
I20250905 02:23:20.430202 22178 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.1:40643
I20250905 02:23:20.430338 22328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.1:40643 every 8 connection(s)
I20250905 02:23:20.433957 22178 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/info.pb
I20250905 02:23:20.439764 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 22178
I20250905 02:23:20.442255 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.2:44381
--local_ip_for_outbound_sockets=127.15.105.2
--tserver_master_addrs=127.15.105.62:34739
--webserver_port=43355
--webserver_interface=127.15.105.2
--builtin_ntp_servers=127.15.105.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 02:23:20.484951 22329 heartbeater.cc:344] Connected to a master server at 127.15.105.62:34739
I20250905 02:23:20.485431 22329 heartbeater.cc:461] Registering TS with master...
I20250905 02:23:20.486584 22329 heartbeater.cc:507] Master 127.15.105.62:34739 requested a full tablet report, sending...
I20250905 02:23:20.491292 22141 ts_manager.cc:194] Registered new tserver with Master: 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1:40643)
I20250905 02:23:20.498692 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.1:45809
I20250905 02:23:20.561535 22222 log.cc:826] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Log is configured to *not* fsync() on all Append() calls
W20250905 02:23:20.767946 22333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:23:20.768486 22333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:23:20.769001 22333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:23:20.800434 22333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:23:20.801299 22333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.2
I20250905 02:23:20.839522 22333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.2:44381
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.105.2
--webserver_port=43355
--tserver_master_addrs=127.15.105.62:34739
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.2
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:23:20.841111 22333 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:23:20.843135 22333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:23:20.857357 22340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:21.502962 22329 heartbeater.cc:499] Master 127.15.105.62:34739 was elected leader, sending a full tablet report...
W20250905 02:23:22.259075 22339 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 22333
W20250905 02:23:22.915722 22339 kernel_stack_watchdog.cc:198] Thread 22333 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 02:23:22.916880 22333 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.057s user 0.626s sys 1.266s
W20250905 02:23:22.917315 22333 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.058s user 0.626s sys 1.266s
W20250905 02:23:20.859822 22341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:22.919193 22343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:22.923502 22342 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2062 milliseconds
I20250905 02:23:22.923519 22333 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:23:22.924775 22333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:23:22.926834 22333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:23:22.928167 22333 hybrid_clock.cc:648] HybridClock initialized: now 1757039002928125 us; error 64 us; skew 500 ppm
I20250905 02:23:22.928920 22333 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:22.935878 22333 webserver.cc:480] Webserver started at http://127.15.105.2:43355/ using document root <none> and password file <none>
I20250905 02:23:22.936879 22333 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:22.937124 22333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:22.944864 22333 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.002s
I20250905 02:23:22.949959 22351 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:22.951022 22333 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250905 02:23:22.951335 22333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
uuid: "f3b55e4b3af944a18c843b4bdc74a7a5"
format_stamp: "Formatted at 2025-09-05 02:23:04 on dist-test-slave-ctw4"
I20250905 02:23:22.953249 22333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:23.006937 22333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:23.008411 22333 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:23.008831 22333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:23.011855 22333 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:23:23.018051 22358 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 02:23:23.025940 22333 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 02:23:23.026155 22333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250905 02:23:23.026429 22333 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 02:23:23.030869 22333 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 02:23:23.031126 22333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250905 02:23:23.031481 22358 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Bootstrap starting.
I20250905 02:23:23.258865 22333 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.2:44381
I20250905 02:23:23.259012 22464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.2:44381 every 8 connection(s)
I20250905 02:23:23.262471 22333 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/info.pb
I20250905 02:23:23.265453 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 22333
I20250905 02:23:23.267549 15780 external_mini_cluster.cc:1366] Running /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
/tmp/dist-test-task4IPHPG/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.105.3:42637
--local_ip_for_outbound_sockets=127.15.105.3
--tserver_master_addrs=127.15.105.62:34739
--webserver_port=36243
--webserver_interface=127.15.105.3
--builtin_ntp_servers=127.15.105.20:33013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 02:23:23.321346 22465 heartbeater.cc:344] Connected to a master server at 127.15.105.62:34739
I20250905 02:23:23.321853 22465 heartbeater.cc:461] Registering TS with master...
I20250905 02:23:23.323338 22465 heartbeater.cc:507] Master 127.15.105.62:34739 requested a full tablet report, sending...
I20250905 02:23:23.328157 22141 ts_manager.cc:194] Registered new tserver with Master: f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381)
I20250905 02:23:23.331135 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.2:57547
I20250905 02:23:23.351383 22358 log.cc:826] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Log is configured to *not* fsync() on all Append() calls
W20250905 02:23:23.770766 22468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 02:23:23.771405 22468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 02:23:23.772214 22468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 02:23:23.827399 22468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 02:23:23.828799 22468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.105.3
I20250905 02:23:23.889602 22468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.105.20:33013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.105.3:42637
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.105.3
--webserver_port=36243
--tserver_master_addrs=127.15.105.62:34739
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.105.3
--log_dir=/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 01:54:52 UTC on 5fd53c4cbb9d
build id 7905
TSAN enabled
I20250905 02:23:23.891270 22468 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:23:23.893381 22468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:23:23.908128 22476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:23.962742 22222 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:23.964172 22222 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Bootstrap complete.
I20250905 02:23:23.966454 22222 ts_tablet_manager.cc:1397] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Time spent bootstrapping tablet: real 3.719s user 3.482s sys 0.092s
I20250905 02:23:23.989888 22222 raft_consensus.cc:357] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:23.994352 22222 raft_consensus.cc:738] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8edf8fb0c74f6fa51f001510404cdc, State: Initialized, Role: FOLLOWER
I20250905 02:23:23.995819 22222 consensus_queue.cc:260] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:24.011732 22222 ts_tablet_manager.cc:1428] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Time spent starting tablet: real 0.045s user 0.039s sys 0.004s
I20250905 02:23:24.336277 22465 heartbeater.cc:499] Master 127.15.105.62:34739 was elected leader, sending a full tablet report...
W20250905 02:23:25.313366 22475 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 22468
W20250905 02:23:25.472674 22468 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.556s user 0.441s sys 0.800s
W20250905 02:23:25.473109 22468 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.557s user 0.441s sys 0.800s
W20250905 02:23:23.918146 22477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:25.474439 22478 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1558 milliseconds
W20250905 02:23:25.475682 22479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:25.475637 22468 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 02:23:25.479198 22468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 02:23:25.481393 22468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 02:23:25.482777 22468 hybrid_clock.cc:648] HybridClock initialized: now 1757039005482732 us; error 55 us; skew 500 ppm
I20250905 02:23:25.483526 22468 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:25.490200 22468 webserver.cc:480] Webserver started at http://127.15.105.3:36243/ using document root <none> and password file <none>
I20250905 02:23:25.491083 22468 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:25.491295 22468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:25.499298 22468 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250905 02:23:25.504315 22487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:25.505354 22468 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250905 02:23:25.505658 22468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data,/tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
uuid: "b76ad76b8be24a269131e185d694499c"
format_stamp: "Formatted at 2025-09-05 02:23:06 on dist-test-slave-ctw4"
I20250905 02:23:25.507495 22468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:25.561647 22468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:25.563082 22468 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:23:25.563524 22468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:25.566152 22468 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:23:25.572801 22494 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 02:23:25.584426 22468 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 02:23:25.584676 22468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.002s
I20250905 02:23:25.584944 22468 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 02:23:25.589670 22468 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 02:23:25.589890 22468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250905 02:23:25.590340 22494 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Bootstrap starting.
I20250905 02:23:25.723085 22555 raft_consensus.cc:491] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:25.723946 22555 raft_consensus.cc:513] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:25.733788 22555 leader_election.cc:290] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381), b76ad76b8be24a269131e185d694499c (127.15.105.3:42637)
W20250905 02:23:25.763415 22216 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.105.3:42637: connect: Connection refused (error 111)
W20250905 02:23:25.773254 22216 leader_election.cc:336] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b76ad76b8be24a269131e185d694499c (127.15.105.3:42637): Network error: Client connection negotiation failed: client connection to 127.15.105.3:42637: connect: Connection refused (error 111)
I20250905 02:23:25.783875 22420 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" is_pre_election: true
W20250905 02:23:25.797065 22219 leader_election.cc:343] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:23:25.797540 22219 leader_election.cc:304] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d8edf8fb0c74f6fa51f001510404cdc; no voters: b76ad76b8be24a269131e185d694499c, f3b55e4b3af944a18c843b4bdc74a7a5
I20250905 02:23:25.798717 22555 raft_consensus.cc:2747] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 02:23:25.860498 22468 rpc_server.cc:307] RPC server started. Bound to: 127.15.105.3:42637
I20250905 02:23:25.860636 22605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.15.105.3:42637 every 8 connection(s)
I20250905 02:23:25.864217 22468 server_base.cc:1179] Dumped server information to /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/info.pb
I20250905 02:23:25.865445 15780 external_mini_cluster.cc:1428] Started /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu as pid 22468
I20250905 02:23:25.914379 22606 heartbeater.cc:344] Connected to a master server at 127.15.105.62:34739
I20250905 02:23:25.914889 22606 heartbeater.cc:461] Registering TS with master...
I20250905 02:23:25.916159 22606 heartbeater.cc:507] Master 127.15.105.62:34739 requested a full tablet report, sending...
I20250905 02:23:25.919852 22141 ts_manager.cc:194] Registered new tserver with Master: b76ad76b8be24a269131e185d694499c (127.15.105.3:42637)
I20250905 02:23:25.922801 22141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.105.3:43709
I20250905 02:23:25.923501 15780 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 02:23:25.963755 22494 log.cc:826] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Log is configured to *not* fsync() on all Append() calls
I20250905 02:23:26.514089 22358 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:26.514900 22358 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Bootstrap complete.
I20250905 02:23:26.516245 22358 ts_tablet_manager.cc:1397] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Time spent bootstrapping tablet: real 3.485s user 3.388s sys 0.059s
I20250905 02:23:26.521332 22358 raft_consensus.cc:357] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:26.523247 22358 raft_consensus.cc:738] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b55e4b3af944a18c843b4bdc74a7a5, State: Initialized, Role: FOLLOWER
I20250905 02:23:26.523996 22358 consensus_queue.cc:260] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:26.527117 22358 ts_tablet_manager.cc:1428] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20250905 02:23:26.926605 22606 heartbeater.cc:499] Master 127.15.105.62:34739 was elected leader, sending a full tablet report...
I20250905 02:23:27.357617 22620 raft_consensus.cc:491] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:27.357986 22620 raft_consensus.cc:513] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:27.359323 22620 leader_election.cc:290] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381), b76ad76b8be24a269131e185d694499c (127.15.105.3:42637)
I20250905 02:23:27.360316 22420 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" is_pre_election: true
I20250905 02:23:27.361042 22420 raft_consensus.cc:2466] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d8edf8fb0c74f6fa51f001510404cdc in term 1.
I20250905 02:23:27.362267 22219 leader_election.cc:304] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8edf8fb0c74f6fa51f001510404cdc, f3b55e4b3af944a18c843b4bdc74a7a5; no voters:
I20250905 02:23:27.363988 22620 raft_consensus.cc:2802] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:23:27.364516 22620 raft_consensus.cc:491] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:27.364972 22620 raft_consensus.cc:3058] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:27.372018 22620 raft_consensus.cc:513] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:27.374465 22620 leader_election.cc:290] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 election: Requested vote from peers f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381), b76ad76b8be24a269131e185d694499c (127.15.105.3:42637)
I20250905 02:23:27.374886 22420 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5"
I20250905 02:23:27.375422 22420 raft_consensus.cc:3058] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:27.376132 22559 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b76ad76b8be24a269131e185d694499c" is_pre_election: true
I20250905 02:23:27.376482 22556 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa1d90e9c09436490ac65d5498ad2ca" candidate_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b76ad76b8be24a269131e185d694499c"
I20250905 02:23:27.383989 22420 raft_consensus.cc:2466] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d8edf8fb0c74f6fa51f001510404cdc in term 2.
I20250905 02:23:27.385136 22219 leader_election.cc:304] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8edf8fb0c74f6fa51f001510404cdc, f3b55e4b3af944a18c843b4bdc74a7a5; no voters:
W20250905 02:23:27.385418 22216 leader_election.cc:343] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b76ad76b8be24a269131e185d694499c (127.15.105.3:42637): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:23:27.385929 22620 raft_consensus.cc:2802] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:23:27.387532 22620 raft_consensus.cc:695] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 2 LEADER]: Becoming Leader. State: Replica: 5d8edf8fb0c74f6fa51f001510404cdc, State: Running, Role: LEADER
I20250905 02:23:27.388473 22620 consensus_queue.cc:237] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:27.399997 22141 catalog_manager.cc:5582] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc reported cstate change: term changed from 0 to 2, leader changed from <none> to 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1), VOTER 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1) added, VOTER b76ad76b8be24a269131e185d694499c (127.15.105.3) added, VOTER f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2) added. New cstate: current_term: 2 leader_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } health_report { overall_health: HEALTHY } } }
W20250905 02:23:27.844033 15780 scanner-internal.cc:458] Time spent opening tablet: real 1.884s user 0.007s sys 0.000s
I20250905 02:23:27.880719 22494 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:27.881821 22494 tablet_bootstrap.cc:492] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Bootstrap complete.
I20250905 02:23:27.884083 22494 ts_tablet_manager.cc:1397] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Time spent bootstrapping tablet: real 2.294s user 2.157s sys 0.076s
I20250905 02:23:27.891898 22494 raft_consensus.cc:357] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:27.894981 22494 raft_consensus.cc:738] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76ad76b8be24a269131e185d694499c, State: Initialized, Role: FOLLOWER
I20250905 02:23:27.895910 22494 consensus_queue.cc:260] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:27.902576 22494 ts_tablet_manager.cc:1428] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20250905 02:23:27.989665 22559 raft_consensus.cc:3058] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:27.996865 22559 raft_consensus.cc:1273] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 2 FOLLOWER]: Refusing update from remote peer 5d8edf8fb0c74f6fa51f001510404cdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250905 02:23:27.998176 22634 consensus_queue.cc:1035] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250905 02:23:28.019361 22420 raft_consensus.cc:1273] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 2 FOLLOWER]: Refusing update from remote peer 5d8edf8fb0c74f6fa51f001510404cdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250905 02:23:28.020998 22634 consensus_queue.cc:1035] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250905 02:23:28.069571 22284 consensus_queue.cc:237] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:28.073406 22559 raft_consensus.cc:1273] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 2 FOLLOWER]: Refusing update from remote peer 5d8edf8fb0c74f6fa51f001510404cdc: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250905 02:23:28.074559 22637 consensus_queue.cc:1035] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250905 02:23:28.079273 22634 raft_consensus.cc:2953] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } } }
I20250905 02:23:28.086984 22559 raft_consensus.cc:2953] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } } }
I20250905 02:23:28.088225 22125 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bfa1d90e9c09436490ac65d5498ad2ca with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:23:28.091521 22141 catalog_manager.cc:5582] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc reported cstate change: config changed from index -1 to 207, VOTER f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2) evicted. New cstate: current_term: 2 leader_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } health_report { overall_health: HEALTHY } } }
I20250905 02:23:28.113277 22284 consensus_queue.cc:237] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:28.116025 22634 raft_consensus.cc:2953] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER b76ad76b8be24a269131e185d694499c (127.15.105.3) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } } }
I20250905 02:23:28.125790 22125 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bfa1d90e9c09436490ac65d5498ad2ca with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:23:28.129081 22140 catalog_manager.cc:5582] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc reported cstate change: config changed from index 207 to 208, VOTER b76ad76b8be24a269131e185d694499c (127.15.105.3) evicted. New cstate: current_term: 2 leader_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } health_report { overall_health: HEALTHY } } }
I20250905 02:23:28.137058 22400 tablet_service.cc:1515] Processing DeleteTablet for tablet bfa1d90e9c09436490ac65d5498ad2ca with delete_type TABLET_DATA_TOMBSTONED (TS f3b55e4b3af944a18c843b4bdc74a7a5 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:38246
I20250905 02:23:28.146643 22649 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:28.147547 22649 raft_consensus.cc:2241] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:28.148243 22649 raft_consensus.cc:2270] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:28.156656 22536 tablet_service.cc:1515] Processing DeleteTablet for tablet bfa1d90e9c09436490ac65d5498ad2ca with delete_type TABLET_DATA_TOMBSTONED (TS b76ad76b8be24a269131e185d694499c not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:45260
I20250905 02:23:28.169872 22651 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:28.170753 22651 raft_consensus.cc:2241] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:28.171460 22651 raft_consensus.cc:2270] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:28.199030 22651 ts_tablet_manager.cc:1905] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:23:28.199151 22649 ts_tablet_manager.cc:1905] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:23:28.214771 22651 ts_tablet_manager.cc:1918] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250905 02:23:28.215276 22651 log.cc:1199] T bfa1d90e9c09436490ac65d5498ad2ca P b76ad76b8be24a269131e185d694499c: Deleting WAL directory at /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/wal/wals/bfa1d90e9c09436490ac65d5498ad2ca
I20250905 02:23:28.217172 22125 catalog_manager.cc:4928] TS b76ad76b8be24a269131e185d694499c (127.15.105.3:42637): tablet bfa1d90e9c09436490ac65d5498ad2ca (table pre_rebuild [id=d32127ef69524701b628c88782ed72b8]) successfully deleted
I20250905 02:23:28.225867 22649 ts_tablet_manager.cc:1918] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250905 02:23:28.226352 22649 log.cc:1199] T bfa1d90e9c09436490ac65d5498ad2ca P f3b55e4b3af944a18c843b4bdc74a7a5: Deleting WAL directory at /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/wal/wals/bfa1d90e9c09436490ac65d5498ad2ca
I20250905 02:23:28.228449 22128 catalog_manager.cc:4928] TS f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381): tablet bfa1d90e9c09436490ac65d5498ad2ca (table pre_rebuild [id=d32127ef69524701b628c88782ed72b8]) successfully deleted
I20250905 02:23:28.595923 22400 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 02:23:28.597088 22264 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 02:23:28.598007 22536 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
fac50144742b4920a90057bedebab6e1 | 127.15.105.62:34739 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.105.20:33013 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
5d8edf8fb0c74f6fa51f001510404cdc | 127.15.105.1:40643 | HEALTHY | <none> | 1 | 0
b76ad76b8be24a269131e185d694499c | 127.15.105.3:42637 | HEALTHY | <none> | 0 | 0
f3b55e4b3af944a18c843b4bdc74a7a5 | 127.15.105.2:44381 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.105.1 | experimental | 127.15.105.1:40643
local_ip_for_outbound_sockets | 127.15.105.2 | experimental | 127.15.105.2:44381
local_ip_for_outbound_sockets | 127.15.105.3 | experimental | 127.15.105.3:42637
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.105.1:40643
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.105.2:44381
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.105.3:42637
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.105.20:33013 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 02:23:28.879760 15780 log_verifier.cc:126] Checking tablet bfa1d90e9c09436490ac65d5498ad2ca
I20250905 02:23:29.187770 15780 log_verifier.cc:177] Verified matching terms for 208 ops in tablet bfa1d90e9c09436490ac65d5498ad2ca
I20250905 02:23:29.190624 22141 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60238:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250905 02:23:29.191231 22141 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60238:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250905 02:23:29.206930 22141 catalog_manager.cc:5869] T 00000000000000000000000000000000 P fac50144742b4920a90057bedebab6e1: Sending DeleteTablet for 1 replicas of tablet bfa1d90e9c09436490ac65d5498ad2ca
I20250905 02:23:29.209120 15780 test_util.cc:276] Using random seed: -2046813433
I20250905 02:23:29.208943 22264 tablet_service.cc:1515] Processing DeleteTablet for tablet bfa1d90e9c09436490ac65d5498ad2ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:29 UTC) from {username='slave'} at 127.0.0.1:47122
I20250905 02:23:29.226804 22681 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:29.227763 22681 raft_consensus.cc:2241] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:23:29.232364 22681 raft_consensus.cc:2270] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:29.260313 22141 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35344:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:23:29.263763 22141 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:23:29.285317 22681 ts_tablet_manager.cc:1905] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:29.301159 22400 tablet_service.cc:1468] Processing CreateTablet for tablet 176f6974f2d649b4bd16a29c2e3439c6 (DEFAULT_TABLE table=post_rebuild [id=8675d61d1d924228a23d2f9a674849cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:23:29.302534 22400 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 176f6974f2d649b4bd16a29c2e3439c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:29.302673 22536 tablet_service.cc:1468] Processing CreateTablet for tablet 176f6974f2d649b4bd16a29c2e3439c6 (DEFAULT_TABLE table=post_rebuild [id=8675d61d1d924228a23d2f9a674849cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:23:29.303911 22536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 176f6974f2d649b4bd16a29c2e3439c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:29.304988 22681 ts_tablet_manager.cc:1918] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250905 02:23:29.305430 22681 log.cc:1199] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Deleting WAL directory at /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/wal/wals/bfa1d90e9c09436490ac65d5498ad2ca
I20250905 02:23:29.306458 22681 ts_tablet_manager.cc:1939] T bfa1d90e9c09436490ac65d5498ad2ca P 5d8edf8fb0c74f6fa51f001510404cdc: Deleting consensus metadata
I20250905 02:23:29.309809 22125 catalog_manager.cc:4928] TS 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1:40643): tablet bfa1d90e9c09436490ac65d5498ad2ca (table pre_rebuild [id=d32127ef69524701b628c88782ed72b8]) successfully deleted
I20250905 02:23:29.319291 22688 tablet_bootstrap.cc:492] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5: Bootstrap starting.
I20250905 02:23:29.323014 22264 tablet_service.cc:1468] Processing CreateTablet for tablet 176f6974f2d649b4bd16a29c2e3439c6 (DEFAULT_TABLE table=post_rebuild [id=8675d61d1d924228a23d2f9a674849cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:23:29.324553 22264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 176f6974f2d649b4bd16a29c2e3439c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:29.328145 22688 tablet_bootstrap.cc:654] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:29.342964 22689 tablet_bootstrap.cc:492] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c: Bootstrap starting.
I20250905 02:23:29.345108 22688 tablet_bootstrap.cc:492] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5: No bootstrap required, opened a new log
I20250905 02:23:29.345592 22688 ts_tablet_manager.cc:1397] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.004s
I20250905 02:23:29.348623 22688 raft_consensus.cc:357] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.349395 22688 raft_consensus.cc:383] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:29.349510 22689 tablet_bootstrap.cc:654] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:29.349735 22688 raft_consensus.cc:738] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b55e4b3af944a18c843b4bdc74a7a5, State: Initialized, Role: FOLLOWER
I20250905 02:23:29.350467 22688 consensus_queue.cc:260] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.354962 22691 tablet_bootstrap.cc:492] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc: Bootstrap starting.
I20250905 02:23:29.358906 22689 tablet_bootstrap.cc:492] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c: No bootstrap required, opened a new log
I20250905 02:23:29.359339 22689 ts_tablet_manager.cc:1397] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20250905 02:23:29.361865 22691 tablet_bootstrap.cc:654] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:29.362165 22689 raft_consensus.cc:357] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.362874 22689 raft_consensus.cc:383] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:29.363194 22689 raft_consensus.cc:738] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76ad76b8be24a269131e185d694499c, State: Initialized, Role: FOLLOWER
I20250905 02:23:29.364112 22689 consensus_queue.cc:260] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.367619 22688 ts_tablet_manager.cc:1428] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5: Time spent starting tablet: real 0.022s user 0.015s sys 0.003s
I20250905 02:23:29.370257 22689 ts_tablet_manager.cc:1428] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20250905 02:23:29.375993 22691 tablet_bootstrap.cc:492] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc: No bootstrap required, opened a new log
I20250905 02:23:29.376458 22691 ts_tablet_manager.cc:1397] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.007s
I20250905 02:23:29.379319 22691 raft_consensus.cc:357] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.380122 22691 raft_consensus.cc:383] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:29.380342 22691 raft_consensus.cc:738] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d8edf8fb0c74f6fa51f001510404cdc, State: Initialized, Role: FOLLOWER
I20250905 02:23:29.380930 22691 consensus_queue.cc:260] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.383142 22691 ts_tablet_manager.cc:1428] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250905 02:23:29.386217 22695 raft_consensus.cc:491] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:29.386628 22695 raft_consensus.cc:513] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.390681 22695 leader_election.cc:290] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381), 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1:40643)
W20250905 02:23:29.411441 22611 tablet.cc:2378] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:23:29.417888 22284 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176f6974f2d649b4bd16a29c2e3439c6" candidate_uuid: "b76ad76b8be24a269131e185d694499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" is_pre_election: true
I20250905 02:23:29.418767 22284 raft_consensus.cc:2466] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b76ad76b8be24a269131e185d694499c in term 0.
I20250905 02:23:29.419924 22488 leader_election.cc:304] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d8edf8fb0c74f6fa51f001510404cdc, b76ad76b8be24a269131e185d694499c; no voters:
I20250905 02:23:29.419878 22420 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176f6974f2d649b4bd16a29c2e3439c6" candidate_uuid: "b76ad76b8be24a269131e185d694499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" is_pre_election: true
I20250905 02:23:29.420444 22420 raft_consensus.cc:2466] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b76ad76b8be24a269131e185d694499c in term 0.
I20250905 02:23:29.420578 22695 raft_consensus.cc:2802] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:29.420840 22695 raft_consensus.cc:491] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:29.421104 22695 raft_consensus.cc:3058] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:29.425616 22695 raft_consensus.cc:513] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.427639 22420 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176f6974f2d649b4bd16a29c2e3439c6" candidate_uuid: "b76ad76b8be24a269131e185d694499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5"
I20250905 02:23:29.427929 22284 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176f6974f2d649b4bd16a29c2e3439c6" candidate_uuid: "b76ad76b8be24a269131e185d694499c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d8edf8fb0c74f6fa51f001510404cdc"
I20250905 02:23:29.428176 22420 raft_consensus.cc:3058] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:29.428462 22284 raft_consensus.cc:3058] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:29.432727 22695 leader_election.cc:290] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [CANDIDATE]: Term 1 election: Requested vote from peers f3b55e4b3af944a18c843b4bdc74a7a5 (127.15.105.2:44381), 5d8edf8fb0c74f6fa51f001510404cdc (127.15.105.1:40643)
I20250905 02:23:29.434716 22420 raft_consensus.cc:2466] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b76ad76b8be24a269131e185d694499c in term 1.
I20250905 02:23:29.435936 22284 raft_consensus.cc:2466] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b76ad76b8be24a269131e185d694499c in term 1.
I20250905 02:23:29.435987 22491 leader_election.cc:304] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b76ad76b8be24a269131e185d694499c, f3b55e4b3af944a18c843b4bdc74a7a5; no voters:
I20250905 02:23:29.436708 22695 raft_consensus.cc:2802] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:29.440572 22695 raft_consensus.cc:695] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [term 1 LEADER]: Becoming Leader. State: Replica: b76ad76b8be24a269131e185d694499c, State: Running, Role: LEADER
I20250905 02:23:29.441465 22695 consensus_queue.cc:237] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } }
I20250905 02:23:29.449443 22141 catalog_manager.cc:5582] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c reported cstate change: term changed from 0 to 1, leader changed from <none> to b76ad76b8be24a269131e185d694499c (127.15.105.3). New cstate: current_term: 1 leader_uuid: "b76ad76b8be24a269131e185d694499c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76ad76b8be24a269131e185d694499c" member_type: VOTER last_known_addr { host: "127.15.105.3" port: 42637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 } health_report { overall_health: UNKNOWN } } }
W20250905 02:23:29.481496 22330 tablet.cc:2378] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 02:23:29.570894 22469 tablet.cc:2378] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:23:29.622455 22420 raft_consensus.cc:1273] T 176f6974f2d649b4bd16a29c2e3439c6 P f3b55e4b3af944a18c843b4bdc74a7a5 [term 1 FOLLOWER]: Refusing update from remote peer b76ad76b8be24a269131e185d694499c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:23:29.622745 22284 raft_consensus.cc:1273] T 176f6974f2d649b4bd16a29c2e3439c6 P 5d8edf8fb0c74f6fa51f001510404cdc [term 1 FOLLOWER]: Refusing update from remote peer b76ad76b8be24a269131e185d694499c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:23:29.624078 22701 consensus_queue.cc:1035] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3b55e4b3af944a18c843b4bdc74a7a5" member_type: VOTER last_known_addr { host: "127.15.105.2" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:23:29.624995 22695 consensus_queue.cc:1035] T 176f6974f2d649b4bd16a29c2e3439c6 P b76ad76b8be24a269131e185d694499c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d8edf8fb0c74f6fa51f001510404cdc" member_type: VOTER last_known_addr { host: "127.15.105.1" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:23:30.067356 22705 meta_cache.cc:1261] Time spent looking up entry by key: real 0.057s user 0.003s sys 0.000s
W20250905 02:23:31.179365 22682 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.15.105.3:42637, user_credentials={real_user=slave}} blocked reactor thread for 43536.2us
I20250905 02:23:34.786165 22264 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 02:23:34.792310 22400 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 02:23:34.792734 22536 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
fac50144742b4920a90057bedebab6e1 | 127.15.105.62:34739 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.105.20:33013 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
5d8edf8fb0c74f6fa51f001510404cdc | 127.15.105.1:40643 | HEALTHY | <none> | 0 | 0
b76ad76b8be24a269131e185d694499c | 127.15.105.3:42637 | HEALTHY | <none> | 1 | 0
f3b55e4b3af944a18c843b4bdc74a7a5 | 127.15.105.2:44381 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.15.105.1 | experimental | 127.15.105.1:40643
local_ip_for_outbound_sockets | 127.15.105.2 | experimental | 127.15.105.2:44381
local_ip_for_outbound_sockets | 127.15.105.3 | experimental | 127.15.105.3:42637
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-0/data/info.pb | hidden | 127.15.105.1:40643
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-1/data/info.pb | hidden | 127.15.105.2:44381
server_dump_info_path | /tmp/dist-test-task4IPHPG/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757038821926622-15780-0/minicluster-data/ts-2/data/info.pb | hidden | 127.15.105.3:42637
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.15.105.20:33013 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 02:23:35.026469 15780 log_verifier.cc:126] Checking tablet 176f6974f2d649b4bd16a29c2e3439c6
I20250905 02:23:35.805434 15780 log_verifier.cc:177] Verified matching terms for 206 ops in tablet 176f6974f2d649b4bd16a29c2e3439c6
I20250905 02:23:35.806227 15780 log_verifier.cc:126] Checking tablet bfa1d90e9c09436490ac65d5498ad2ca
I20250905 02:23:35.806471 15780 log_verifier.cc:177] Verified matching terms for 0 ops in tablet bfa1d90e9c09436490ac65d5498ad2ca
I20250905 02:23:35.834822 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 22178
I20250905 02:23:35.880730 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 22333
I20250905 02:23:35.921677 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 22468
I20250905 02:23:35.967403 15780 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task4IPHPG/build/tsan/bin/kudu with pid 22108
2025-09-05T02:23:35Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (37695 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (37695 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (194019 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST
I20250905 02:23:36.040009 15780 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 1 messages since previous log ~49 seconds ago