Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250904 19:57:55.456571 2776 test_util.cc:276] Using random seed: 589237699
W20250904 19:57:56.587524 2776 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.091s user 0.416s sys 0.675s
W20250904 19:57:56.587852 2776 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.092s user 0.416s sys 0.675s
I20250904 19:57:56.591935 2776 ts_itest-base.cc:115] Starting cluster with:
I20250904 19:57:56.592159 2776 ts_itest-base.cc:116] --------------
I20250904 19:57:56.592348 2776 ts_itest-base.cc:117] 4 tablet servers
I20250904 19:57:56.592532 2776 ts_itest-base.cc:118] 3 replicas per TS
I20250904 19:57:56.592700 2776 ts_itest-base.cc:119] --------------
2025-09-04T19:57:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T19:57:56Z Disabled control of system clock
I20250904 19:57:56.627115 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46421
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:46421 with env {}
W20250904 19:57:56.925757 2790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:57:56.926398 2790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:57:56.926895 2790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:57:56.959136 2790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:57:56.959458 2790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:57:56.959663 2790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:57:56.959862 2790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:57:56.996531 2790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:46421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:57:56.997720 2790 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:57:56.999313 2790 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:57:57.009588 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:57:57.010803 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:57:57.013737 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:57:58.109776 2798 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250904 19:57:58.109807 2790 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:57:58.113399 2790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:57:58.116014 2790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:57:58.117460 2790 hybrid_clock.cc:648] HybridClock initialized: now 1757015878117387 us; error 79 us; skew 500 ppm
I20250904 19:57:58.118451 2790 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:57:58.124023 2790 webserver.cc:480] Webserver started at http://127.2.182.62:41803/ using document root <none> and password file <none>
I20250904 19:57:58.124910 2790 fs_manager.cc:362] Metadata directory not provided
I20250904 19:57:58.125113 2790 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:57:58.125531 2790 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:57:58.129756 2790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "12b8ec2445d9472c9ac8f8a3c7488b88"
format_stamp: "Formatted at 2025-09-04 19:57:58 on dist-test-slave-tjsx"
I20250904 19:57:58.130856 2790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "12b8ec2445d9472c9ac8f8a3c7488b88"
format_stamp: "Formatted at 2025-09-04 19:57:58 on dist-test-slave-tjsx"
I20250904 19:57:58.137622 2790 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250904 19:57:58.142917 2806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:57:58.143860 2790 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250904 19:57:58.144178 2790 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "12b8ec2445d9472c9ac8f8a3c7488b88"
format_stamp: "Formatted at 2025-09-04 19:57:58 on dist-test-slave-tjsx"
I20250904 19:57:58.144500 2790 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:57:58.191184 2790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:57:58.192600 2790 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:57:58.193043 2790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:57:58.259160 2790 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:46421
I20250904 19:57:58.259213 2857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:46421 every 8 connection(s)
I20250904 19:57:58.261740 2790 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:57:58.265503 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 2790
I20250904 19:57:58.266002 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250904 19:57:58.267305 2858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:57:58.286056 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Bootstrap starting.
I20250904 19:57:58.292192 2858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Neither blocks nor log segments found. Creating new log.
I20250904 19:57:58.294377 2858 log.cc:826] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Log is configured to *not* fsync() on all Append() calls
I20250904 19:57:58.299757 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: No bootstrap required, opened a new log
I20250904 19:57:58.318158 2858 raft_consensus.cc:357] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } }
I20250904 19:57:58.318831 2858 raft_consensus.cc:383] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:57:58.319113 2858 raft_consensus.cc:738] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b8ec2445d9472c9ac8f8a3c7488b88, State: Initialized, Role: FOLLOWER
I20250904 19:57:58.319852 2858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } }
I20250904 19:57:58.320336 2858 raft_consensus.cc:397] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:57:58.320598 2858 raft_consensus.cc:491] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:57:58.320873 2858 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:57:58.325013 2858 raft_consensus.cc:513] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } }
I20250904 19:57:58.325673 2858 leader_election.cc:304] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12b8ec2445d9472c9ac8f8a3c7488b88; no voters:
I20250904 19:57:58.327289 2858 leader_election.cc:290] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:57:58.327915 2863 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:57:58.330479 2863 raft_consensus.cc:695] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 1 LEADER]: Becoming Leader. State: Replica: 12b8ec2445d9472c9ac8f8a3c7488b88, State: Running, Role: LEADER
I20250904 19:57:58.331274 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } }
I20250904 19:57:58.331712 2858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:57:58.341164 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } } }
I20250904 19:57:58.341878 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: This master's current role is: LEADER
I20250904 19:57:58.344009 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12b8ec2445d9472c9ac8f8a3c7488b88. Latest consensus state: current_term: 1 leader_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } } }
I20250904 19:57:58.344631 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: This master's current role is: LEADER
I20250904 19:57:58.351662 2871 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:57:58.363121 2871 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:57:58.377700 2871 catalog_manager.cc:1349] Generated new cluster ID: 7825a03f825e4b968ac2b31b2998b610
I20250904 19:57:58.377954 2871 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:57:58.396908 2871 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 19:57:58.398263 2871 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:57:58.413913 2871 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Generated new TSK 0
I20250904 19:57:58.414970 2871 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 19:57:58.432754 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46421
--builtin_ntp_servers=127.2.182.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250904 19:57:58.718592 2882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:57:58.719089 2882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:57:58.719560 2882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:57:58.750017 2882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:57:58.750880 2882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 19:57:58.784883 2882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:57:58.786088 2882 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:57:58.787626 2882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:57:58.799086 2888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:00.201582 2887 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 2882
W20250904 19:58:00.252038 2882 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.454s user 0.571s sys 0.883s
W20250904 19:57:58.800043 2889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:00.252365 2882 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.454s user 0.571s sys 0.883s
I20250904 19:58:00.259709 2882 server_base.cc:1047] running on GCE node
W20250904 19:58:00.261067 2893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:00.262128 2882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:00.264215 2882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:00.265581 2882 hybrid_clock.cc:648] HybridClock initialized: now 1757015880265526 us; error 44 us; skew 500 ppm
I20250904 19:58:00.266571 2882 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:00.271912 2882 webserver.cc:480] Webserver started at http://127.2.182.1:34313/ using document root <none> and password file <none>
I20250904 19:58:00.272773 2882 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:00.272982 2882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:00.273391 2882 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:00.277606 2882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cfd8a6ead99048f5beac07b5ee5a2710"
format_stamp: "Formatted at 2025-09-04 19:58:00 on dist-test-slave-tjsx"
I20250904 19:58:00.278628 2882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cfd8a6ead99048f5beac07b5ee5a2710"
format_stamp: "Formatted at 2025-09-04 19:58:00 on dist-test-slave-tjsx"
I20250904 19:58:00.285459 2882 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250904 19:58:00.290539 2898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:00.291460 2882 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 19:58:00.291745 2882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cfd8a6ead99048f5beac07b5ee5a2710"
format_stamp: "Formatted at 2025-09-04 19:58:00 on dist-test-slave-tjsx"
I20250904 19:58:00.292042 2882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:00.343739 2882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:00.345052 2882 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:00.345446 2882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:00.347771 2882 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:00.351516 2882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:00.351720 2882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:00.351958 2882 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:00.352115 2882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:00.473696 2882 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:46703
I20250904 19:58:00.473798 3010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:46703 every 8 connection(s)
I20250904 19:58:00.476121 2882 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 19:58:00.478437 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 2882
I20250904 19:58:00.478914 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250904 19:58:00.484937 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46421
--builtin_ntp_servers=127.2.182.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:58:00.496651 3011 heartbeater.cc:344] Connected to a master server at 127.2.182.62:46421
I20250904 19:58:00.497018 3011 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:00.498016 3011 heartbeater.cc:507] Master 127.2.182.62:46421 requested a full tablet report, sending...
I20250904 19:58:00.500618 2823 ts_manager.cc:194] Registered new tserver with Master: cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1:46703)
I20250904 19:58:00.503278 2823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:40167
W20250904 19:58:00.764737 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:00.765154 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:00.765595 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:00.795838 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:00.796597 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 19:58:00.830205 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:00.831362 3015 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:00.832790 3015 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:00.843582 3021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:01.506484 3011 heartbeater.cc:499] Master 127.2.182.62:46421 was elected leader, sending a full tablet report...
W20250904 19:58:00.844007 3022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:02.116935 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:02.119879 3023 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1271 milliseconds
W20250904 19:58:02.119915 3015 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.276s user 0.504s sys 0.769s
W20250904 19:58:02.120239 3015 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.277s user 0.504s sys 0.769s
I20250904 19:58:02.120512 3015 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:02.121456 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:02.127188 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:02.128521 3015 hybrid_clock.cc:648] HybridClock initialized: now 1757015882128489 us; error 37 us; skew 500 ppm
I20250904 19:58:02.129282 3015 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:02.136874 3015 webserver.cc:480] Webserver started at http://127.2.182.2:39479/ using document root <none> and password file <none>
I20250904 19:58:02.137723 3015 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:02.137928 3015 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:02.138347 3015 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:02.142566 3015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "daeded77000f4db39e0a8a9a764d0339"
format_stamp: "Formatted at 2025-09-04 19:58:02 on dist-test-slave-tjsx"
I20250904 19:58:02.143636 3015 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "daeded77000f4db39e0a8a9a764d0339"
format_stamp: "Formatted at 2025-09-04 19:58:02 on dist-test-slave-tjsx"
I20250904 19:58:02.150983 3015 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250904 19:58:02.156939 3031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:02.158000 3015 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250904 19:58:02.158321 3015 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "daeded77000f4db39e0a8a9a764d0339"
format_stamp: "Formatted at 2025-09-04 19:58:02 on dist-test-slave-tjsx"
I20250904 19:58:02.158619 3015 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:02.226069 3015 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:02.227577 3015 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:02.228004 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:02.230472 3015 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:02.234552 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:02.234786 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250904 19:58:02.235045 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:02.235280 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:02.361217 3015 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:46295
I20250904 19:58:02.361305 3143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:46295 every 8 connection(s)
I20250904 19:58:02.363598 3015 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250904 19:58:02.366555 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 3015
I20250904 19:58:02.367116 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250904 19:58:02.373365 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46421
--builtin_ntp_servers=127.2.182.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:58:02.383970 3144 heartbeater.cc:344] Connected to a master server at 127.2.182.62:46421
I20250904 19:58:02.384439 3144 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:02.385640 3144 heartbeater.cc:507] Master 127.2.182.62:46421 requested a full tablet report, sending...
I20250904 19:58:02.387912 2823 ts_manager.cc:194] Registered new tserver with Master: daeded77000f4db39e0a8a9a764d0339 (127.2.182.2:46295)
I20250904 19:58:02.389137 2823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:54103
W20250904 19:58:02.668372 3148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:02.668833 3148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:02.669291 3148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:02.700889 3148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:02.701793 3148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 19:58:02.735937 3148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:02.737172 3148 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:02.738744 3148 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:02.749843 3154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:03.392593 3144 heartbeater.cc:499] Master 127.2.182.62:46421 was elected leader, sending a full tablet report...
W20250904 19:58:02.751159 3155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:03.844877 3157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:03.847114 3156 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1092 milliseconds
I20250904 19:58:03.847249 3148 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:03.848281 3148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:03.850736 3148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:03.852121 3148 hybrid_clock.cc:648] HybridClock initialized: now 1757015883852091 us; error 30 us; skew 500 ppm
I20250904 19:58:03.852830 3148 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:03.858621 3148 webserver.cc:480] Webserver started at http://127.2.182.3:40629/ using document root <none> and password file <none>
I20250904 19:58:03.859490 3148 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:03.859694 3148 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:03.860090 3148 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:03.864279 3148 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6bd9b70c6b93412e992a48dceae23d6f"
format_stamp: "Formatted at 2025-09-04 19:58:03 on dist-test-slave-tjsx"
I20250904 19:58:03.865283 3148 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6bd9b70c6b93412e992a48dceae23d6f"
format_stamp: "Formatted at 2025-09-04 19:58:03 on dist-test-slave-tjsx"
I20250904 19:58:03.872310 3148 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250904 19:58:03.877516 3164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:03.878449 3148 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250904 19:58:03.878801 3148 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6bd9b70c6b93412e992a48dceae23d6f"
format_stamp: "Formatted at 2025-09-04 19:58:03 on dist-test-slave-tjsx"
I20250904 19:58:03.879123 3148 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:03.934077 3148 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:03.935498 3148 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:03.935904 3148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:03.938259 3148 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:03.942060 3148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:03.942270 3148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:03.942499 3148 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:03.942651 3148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:04.065164 3148 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:39175
I20250904 19:58:04.065250 3276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:39175 every 8 connection(s)
I20250904 19:58:04.067611 3148 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250904 19:58:04.074594 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 3148
I20250904 19:58:04.075006 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250904 19:58:04.080631 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.4:0
--local_ip_for_outbound_sockets=127.2.182.4
--webserver_interface=127.2.182.4
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46421
--builtin_ntp_servers=127.2.182.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:58:04.090790 3277 heartbeater.cc:344] Connected to a master server at 127.2.182.62:46421
I20250904 19:58:04.091257 3277 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:04.092420 3277 heartbeater.cc:507] Master 127.2.182.62:46421 requested a full tablet report, sending...
I20250904 19:58:04.094660 2823 ts_manager.cc:194] Registered new tserver with Master: 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3:39175)
I20250904 19:58:04.095836 2823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:37789
W20250904 19:58:04.362489 3281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:04.362977 3281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:04.363469 3281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:04.393873 3281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:04.394758 3281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.4
I20250904 19:58:04.428795 3281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.182.4
--webserver_port=0
--tserver_master_addrs=127.2.182.62:46421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.4
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:04.430001 3281 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:04.431543 3281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:04.443084 3287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:05.099321 3277 heartbeater.cc:499] Master 127.2.182.62:46421 was elected leader, sending a full tablet report...
W20250904 19:58:04.443672 3288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:05.522223 3290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:05.524317 3289 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1076 milliseconds
I20250904 19:58:05.524469 3281 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:05.525686 3281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:05.527792 3281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:05.529109 3281 hybrid_clock.cc:648] HybridClock initialized: now 1757015885529068 us; error 43 us; skew 500 ppm
I20250904 19:58:05.529826 3281 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:05.535437 3281 webserver.cc:480] Webserver started at http://127.2.182.4:34493/ using document root <none> and password file <none>
I20250904 19:58:05.536288 3281 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:05.536520 3281 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:05.536948 3281 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:05.541126 3281 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4"
format_stamp: "Formatted at 2025-09-04 19:58:05 on dist-test-slave-tjsx"
I20250904 19:58:05.542140 3281 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4"
format_stamp: "Formatted at 2025-09-04 19:58:05 on dist-test-slave-tjsx"
I20250904 19:58:05.548743 3281 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250904 19:58:05.553884 3298 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:05.554785 3281 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 19:58:05.555073 3281 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4"
format_stamp: "Formatted at 2025-09-04 19:58:05 on dist-test-slave-tjsx"
I20250904 19:58:05.555377 3281 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:05.604816 3281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:05.606148 3281 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:05.606545 3281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:05.608881 3281 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:05.612689 3281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:05.612890 3281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:05.613109 3281 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:05.613268 3281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:05.734627 3281 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.4:46277
I20250904 19:58:05.734767 3410 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.4:46277 every 8 connection(s)
I20250904 19:58:05.737121 3281 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250904 19:58:05.740238 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 3281
I20250904 19:58:05.740922 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250904 19:58:05.756537 3411 heartbeater.cc:344] Connected to a master server at 127.2.182.62:46421
I20250904 19:58:05.756889 3411 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:05.757766 3411 heartbeater.cc:507] Master 127.2.182.62:46421 requested a full tablet report, sending...
I20250904 19:58:05.759814 2823 ts_manager.cc:194] Registered new tserver with Master: 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4:46277)
I20250904 19:58:05.760936 2823 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.4:35297
I20250904 19:58:05.761503 2776 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250904 19:58:05.797221 2823 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41934:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250904 19:58:05.863144 3212 tablet_service.cc:1468] Processing CreateTablet for tablet 520b25df37424f83aef6dd893db557e7 (DEFAULT_TABLE table=TestTable [id=f570ca18124d46168347a0df553d9dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:05.864805 3212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520b25df37424f83aef6dd893db557e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:05.869199 3346 tablet_service.cc:1468] Processing CreateTablet for tablet 520b25df37424f83aef6dd893db557e7 (DEFAULT_TABLE table=TestTable [id=f570ca18124d46168347a0df553d9dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:05.870982 3346 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520b25df37424f83aef6dd893db557e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:05.871317 3079 tablet_service.cc:1468] Processing CreateTablet for tablet 520b25df37424f83aef6dd893db557e7 (DEFAULT_TABLE table=TestTable [id=f570ca18124d46168347a0df553d9dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:05.873008 3079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520b25df37424f83aef6dd893db557e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:05.890564 3430 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f: Bootstrap starting.
I20250904 19:58:05.892671 3431 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4: Bootstrap starting.
I20250904 19:58:05.896271 3430 tablet_bootstrap.cc:654] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:05.898080 3432 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: Bootstrap starting.
I20250904 19:58:05.899549 3431 tablet_bootstrap.cc:654] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:05.899979 3430 log.cc:826] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:05.901345 3431 log.cc:826] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:05.904778 3432 tablet_bootstrap.cc:654] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:05.905364 3430 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f: No bootstrap required, opened a new log
I20250904 19:58:05.905829 3430 ts_tablet_manager.cc:1397] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.013s
I20250904 19:58:05.906975 3432 log.cc:826] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:05.907651 3431 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4: No bootstrap required, opened a new log
I20250904 19:58:05.907995 3431 ts_tablet_manager.cc:1397] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20250904 19:58:05.913094 3432 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: No bootstrap required, opened a new log
I20250904 19:58:05.913594 3432 ts_tablet_manager.cc:1397] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250904 19:58:05.929905 3432 raft_consensus.cc:357] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:05.930931 3432 raft_consensus.cc:383] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:05.931257 3432 raft_consensus.cc:738] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daeded77000f4db39e0a8a9a764d0339, State: Initialized, Role: FOLLOWER
I20250904 19:58:05.931448 3430 raft_consensus.cc:357] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:05.932284 3430 raft_consensus.cc:383] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:05.932004 3431 raft_consensus.cc:357] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:05.932142 3432 consensus_queue.cc:260] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:05.932658 3430 raft_consensus.cc:738] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd9b70c6b93412e992a48dceae23d6f, State: Initialized, Role: FOLLOWER
I20250904 19:58:05.932801 3431 raft_consensus.cc:383] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:05.933109 3431 raft_consensus.cc:738] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ad5475edce46b6bbc61ecbdc5b5ad4, State: Initialized, Role: FOLLOWER
I20250904 19:58:05.933473 3430 consensus_queue.cc:260] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:05.934078 3431 consensus_queue.cc:260] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:05.936208 3432 ts_tablet_manager.cc:1428] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: Time spent starting tablet: real 0.022s user 0.021s sys 0.000s
I20250904 19:58:05.937928 3411 heartbeater.cc:499] Master 127.2.182.62:46421 was elected leader, sending a full tablet report...
I20250904 19:58:05.939205 3430 ts_tablet_manager.cc:1428] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f: Time spent starting tablet: real 0.033s user 0.030s sys 0.000s
I20250904 19:58:05.941637 3431 ts_tablet_manager.cc:1428] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4: Time spent starting tablet: real 0.033s user 0.022s sys 0.008s
W20250904 19:58:05.991175 3412 tablet.cc:2378] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 19:58:06.073292 3278 tablet.cc:2378] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 19:58:06.122262 3145 tablet.cc:2378] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:06.178795 3438 raft_consensus.cc:491] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:58:06.179247 3438 raft_consensus.cc:513] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:06.181449 3438 leader_election.cc:290] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3:39175), daeded77000f4db39e0a8a9a764d0339 (127.2.182.2:46295)
I20250904 19:58:06.189985 3232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd9b70c6b93412e992a48dceae23d6f" is_pre_election: true
I20250904 19:58:06.190634 3232 raft_consensus.cc:2466] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ad5475edce46b6bbc61ecbdc5b5ad4 in term 0.
I20250904 19:58:06.190618 3099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daeded77000f4db39e0a8a9a764d0339" is_pre_election: true
I20250904 19:58:06.191322 3099 raft_consensus.cc:2466] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ad5475edce46b6bbc61ecbdc5b5ad4 in term 0.
I20250904 19:58:06.191651 3302 leader_election.cc:304] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd9b70c6b93412e992a48dceae23d6f, 98ad5475edce46b6bbc61ecbdc5b5ad4; no voters:
I20250904 19:58:06.192344 3438 raft_consensus.cc:2802] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 19:58:06.192590 3438 raft_consensus.cc:491] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 19:58:06.192831 3438 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:06.196920 3438 raft_consensus.cc:513] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:06.198156 3438 leader_election.cc:290] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3:39175), daeded77000f4db39e0a8a9a764d0339 (127.2.182.2:46295)
I20250904 19:58:06.198812 3232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd9b70c6b93412e992a48dceae23d6f"
I20250904 19:58:06.199005 3099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daeded77000f4db39e0a8a9a764d0339"
I20250904 19:58:06.199306 3232 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:06.199479 3099 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:06.205631 3099 raft_consensus.cc:2466] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ad5475edce46b6bbc61ecbdc5b5ad4 in term 1.
I20250904 19:58:06.205629 3232 raft_consensus.cc:2466] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ad5475edce46b6bbc61ecbdc5b5ad4 in term 1.
I20250904 19:58:06.206543 3301 leader_election.cc:304] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98ad5475edce46b6bbc61ecbdc5b5ad4, daeded77000f4db39e0a8a9a764d0339; no voters:
I20250904 19:58:06.207129 3438 raft_consensus.cc:2802] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:06.208490 3438 raft_consensus.cc:695] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [term 1 LEADER]: Becoming Leader. State: Replica: 98ad5475edce46b6bbc61ecbdc5b5ad4, State: Running, Role: LEADER
I20250904 19:58:06.209256 3438 consensus_queue.cc:237] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:06.219213 2823 catalog_manager.cc:5582] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4). New cstate: current_term: 1 leader_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } health_report { overall_health: UNKNOWN } } }
I20250904 19:58:06.318235 2776 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250904 19:58:06.321511 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver daeded77000f4db39e0a8a9a764d0339 to finish bootstrapping
I20250904 19:58:06.333756 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6bd9b70c6b93412e992a48dceae23d6f to finish bootstrapping
I20250904 19:58:06.343199 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 98ad5475edce46b6bbc61ecbdc5b5ad4 to finish bootstrapping
I20250904 19:58:06.352413 2776 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250904 19:58:06.355268 2776 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "520b25df37424f83aef6dd893db557e7"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f"
rpc_addresses {
host: "127.2.182.3"
port: 39175
}
}
ts_infos {
permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4"
rpc_addresses {
host: "127.2.182.4"
port: 46277
}
}
ts_infos {
permanent_uuid: "daeded77000f4db39e0a8a9a764d0339"
rpc_addresses {
host: "127.2.182.2"
port: 46295
}
}
I20250904 19:58:06.770202 3444 consensus_queue.cc:1035] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 19:58:06.787400 3447 consensus_queue.cc:1035] T 520b25df37424f83aef6dd893db557e7 P 98ad5475edce46b6bbc61ecbdc5b5ad4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 19:58:06.795502 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 3281
W20250904 19:58:06.819830 3032 connection.cc:537] server connection from 127.2.182.4:47425 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250904 19:58:06.821131 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 2790
I20250904 19:58:06.844898 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46421
--webserver_interface=127.2.182.62
--webserver_port=41803
--builtin_ntp_servers=127.2.182.20:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:46421 with env {}
W20250904 19:58:07.129889 3455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:07.130450 3455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:07.130914 3455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:07.162804 3455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:58:07.163117 3455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:07.163358 3455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:58:07.163585 3455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:58:07.198104 3455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:46421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:46421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=41803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:07.199343 3455 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:07.200861 3455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:07.211150 3461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:07.546293 3011 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:46421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:46421: connect: Connection refused (error 111)
W20250904 19:58:07.816694 3277 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:46421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:46421: connect: Connection refused (error 111)
W20250904 19:58:07.819522 3144 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:46421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:46421: connect: Connection refused (error 111)
I20250904 19:58:08.299715 3471 raft_consensus.cc:491] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98ad5475edce46b6bbc61ecbdc5b5ad4)
I20250904 19:58:08.301640 3471 raft_consensus.cc:513] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:08.302603 3470 raft_consensus.cc:491] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98ad5475edce46b6bbc61ecbdc5b5ad4)
I20250904 19:58:08.303179 3470 raft_consensus.cc:513] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:08.320922 3471 leader_election.cc:290] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3:39175), 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4:46277)
I20250904 19:58:08.324149 3470 leader_election.cc:290] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4:46277), daeded77000f4db39e0a8a9a764d0339 (127.2.182.2:46295)
W20250904 19:58:08.336463 3035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.4:46277: connect: Connection refused (error 111)
W20250904 19:58:08.341470 3168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.4:46277: connect: Connection refused (error 111)
W20250904 19:58:08.343998 3035 leader_election.cc:336] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4:46277): Network error: Client connection negotiation failed: client connection to 127.2.182.4:46277: connect: Connection refused (error 111)
I20250904 19:58:08.350889 3099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "6bd9b70c6b93412e992a48dceae23d6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "daeded77000f4db39e0a8a9a764d0339" is_pre_election: true
I20250904 19:58:08.351527 3099 raft_consensus.cc:2466] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bd9b70c6b93412e992a48dceae23d6f in term 1.
I20250904 19:58:08.353040 3167 leader_election.cc:304] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd9b70c6b93412e992a48dceae23d6f, daeded77000f4db39e0a8a9a764d0339; no voters:
I20250904 19:58:08.354169 3470 raft_consensus.cc:2802] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 19:58:08.354552 3470 raft_consensus.cc:491] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 98ad5475edce46b6bbc61ecbdc5b5ad4)
I20250904 19:58:08.354965 3470 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 1 FOLLOWER]: Advancing to term 2
W20250904 19:58:08.368918 3168 leader_election.cc:336] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4:46277): Network error: Client connection negotiation failed: client connection to 127.2.182.4:46277: connect: Connection refused (error 111)
I20250904 19:58:08.361801 3470 raft_consensus.cc:513] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
W20250904 19:58:08.376233 3168 leader_election.cc:336] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4:46277): Network error: Client connection negotiation failed: client connection to 127.2.182.4:46277: connect: Connection refused (error 111)
I20250904 19:58:08.383939 3099 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "6bd9b70c6b93412e992a48dceae23d6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "daeded77000f4db39e0a8a9a764d0339"
I20250904 19:58:08.384524 3099 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 1 FOLLOWER]: Advancing to term 2
I20250904 19:58:08.391216 3099 raft_consensus.cc:2466] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bd9b70c6b93412e992a48dceae23d6f in term 2.
I20250904 19:58:08.392349 3167 leader_election.cc:304] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bd9b70c6b93412e992a48dceae23d6f, daeded77000f4db39e0a8a9a764d0339; no voters: 98ad5475edce46b6bbc61ecbdc5b5ad4
I20250904 19:58:08.401321 3232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "daeded77000f4db39e0a8a9a764d0339" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6bd9b70c6b93412e992a48dceae23d6f" is_pre_election: true
I20250904 19:58:08.402112 3232 raft_consensus.cc:2391] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate daeded77000f4db39e0a8a9a764d0339 in current term 2: Already voted for candidate 6bd9b70c6b93412e992a48dceae23d6f in this term.
I20250904 19:58:08.403021 3470 leader_election.cc:290] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [CANDIDATE]: Term 2 election: Requested vote from peers 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4:46277), daeded77000f4db39e0a8a9a764d0339 (127.2.182.2:46295)
I20250904 19:58:08.403364 3035 leader_election.cc:304] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: daeded77000f4db39e0a8a9a764d0339; no voters: 6bd9b70c6b93412e992a48dceae23d6f, 98ad5475edce46b6bbc61ecbdc5b5ad4
I20250904 19:58:08.404078 3470 raft_consensus.cc:2802] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 2 FOLLOWER]: Leader election won for term 2
I20250904 19:58:08.404505 3471 raft_consensus.cc:2747] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250904 19:58:08.406219 3470 raft_consensus.cc:695] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 2 LEADER]: Becoming Leader. State: Replica: 6bd9b70c6b93412e992a48dceae23d6f, State: Running, Role: LEADER
I20250904 19:58:08.407395 3470 consensus_queue.cc:237] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
W20250904 19:58:08.613384 3460 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 3455
W20250904 19:58:08.753134 3455 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.541s user 0.667s sys 0.833s
W20250904 19:58:07.211963 3462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:08.753520 3455 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.542s user 0.667s sys 0.833s
W20250904 19:58:08.755347 3464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:08.758366 3463 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1542 milliseconds
I20250904 19:58:08.758435 3455 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:08.759668 3455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:08.761927 3455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:08.763257 3455 hybrid_clock.cc:648] HybridClock initialized: now 1757015888763212 us; error 39 us; skew 500 ppm
I20250904 19:58:08.764045 3455 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:08.769878 3455 webserver.cc:480] Webserver started at http://127.2.182.62:41803/ using document root <none> and password file <none>
I20250904 19:58:08.770794 3455 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:08.771021 3455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:08.778560 3455 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250904 19:58:08.782716 3485 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:08.783638 3455 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250904 19:58:08.783959 3455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "12b8ec2445d9472c9ac8f8a3c7488b88"
format_stamp: "Formatted at 2025-09-04 19:57:58 on dist-test-slave-tjsx"
I20250904 19:58:08.785871 3455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:08.830020 3455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:08.831548 3455 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:08.831945 3455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:08.896977 3455 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:46421
I20250904 19:58:08.897023 3536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:46421 every 8 connection(s)
I20250904 19:58:08.899711 3455 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:58:08.909201 3537 sys_catalog.cc:263] Verifying existing consensus state
I20250904 19:58:08.909438 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 3455
I20250904 19:58:08.910007 2776 kudu-admin-test.cc:735] Forcing unsafe config change on tserver daeded77000f4db39e0a8a9a764d0339
I20250904 19:58:08.915603 3537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Bootstrap starting.
I20250904 19:58:08.940963 3099 raft_consensus.cc:1273] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 2 FOLLOWER]: Refusing update from remote peer 6bd9b70c6b93412e992a48dceae23d6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250904 19:58:08.942349 3540 consensus_queue.cc:1035] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250904 19:58:08.957916 3537 log.cc:826] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:08.986393 3144 heartbeater.cc:344] Connected to a master server at 127.2.182.62:46421
I20250904 19:58:08.996048 3537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:58:08.997051 3537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Bootstrap complete.
W20250904 19:58:09.006548 3168 consensus_peers.cc:489] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f -> Peer 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4:46277): Couldn't send request to peer 98ad5475edce46b6bbc61ecbdc5b5ad4. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.4:46277: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 19:58:09.007951 3277 heartbeater.cc:344] Connected to a master server at 127.2.182.62:46421
I20250904 19:58:09.024330 3537 raft_consensus.cc:357] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } }
I20250904 19:58:09.027410 3537 raft_consensus.cc:738] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b8ec2445d9472c9ac8f8a3c7488b88, State: Initialized, Role: FOLLOWER
I20250904 19:58:09.028291 3537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } }
I20250904 19:58:09.028977 3537 raft_consensus.cc:397] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:58:09.029345 3537 raft_consensus.cc:491] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:58:09.029712 3537 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 1 FOLLOWER]: Advancing to term 2
I20250904 19:58:09.038112 3537 raft_consensus.cc:513] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } }
I20250904 19:58:09.039007 3537 leader_election.cc:304] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12b8ec2445d9472c9ac8f8a3c7488b88; no voters:
I20250904 19:58:09.040962 3537 leader_election.cc:290] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250904 19:58:09.041297 3551 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 19:58:09.044142 3551 raft_consensus.cc:695] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [term 2 LEADER]: Becoming Leader. State: Replica: 12b8ec2445d9472c9ac8f8a3c7488b88, State: Running, Role: LEADER
I20250904 19:58:09.044904 3551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } }
I20250904 19:58:09.045596 3537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:58:09.053231 3553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12b8ec2445d9472c9ac8f8a3c7488b88. Latest consensus state: current_term: 2 leader_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } } }
I20250904 19:58:09.054315 3552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b8ec2445d9472c9ac8f8a3c7488b88" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 46421 } } }
I20250904 19:58:09.055043 3552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:09.056962 3553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88 [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:09.069716 3558 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:58:09.091114 3558 catalog_manager.cc:671] Loaded metadata for table TestTable [id=f570ca18124d46168347a0df553d9dc6]
I20250904 19:58:09.106506 3558 tablet_loader.cc:96] loaded metadata for tablet 520b25df37424f83aef6dd893db557e7 (table TestTable [id=f570ca18124d46168347a0df553d9dc6])
I20250904 19:58:09.107977 3558 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:58:09.112418 3558 catalog_manager.cc:1261] Loaded cluster ID: 7825a03f825e4b968ac2b31b2998b610
I20250904 19:58:09.112756 3558 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:58:09.119678 3558 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:58:09.124022 3558 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 12b8ec2445d9472c9ac8f8a3c7488b88: Loaded TSK: 0
I20250904 19:58:09.126505 3558 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250904 19:58:09.268862 3539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:09.269441 3539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:09.300091 3539 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250904 19:58:09.617470 3011 heartbeater.cc:344] Connected to a master server at 127.2.182.62:46421
I20250904 19:58:09.623208 3502 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" instance_seqno: 1757015880445813) as {username='slave'} at 127.2.182.1:34921; Asking this server to re-register.
I20250904 19:58:09.624722 3011 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:09.625386 3011 heartbeater.cc:507] Master 127.2.182.62:46421 requested a full tablet report, sending...
I20250904 19:58:09.627852 3502 ts_manager.cc:194] Registered new tserver with Master: cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1:46703)
I20250904 19:58:09.996769 3502 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" instance_seqno: 1757015882333334) as {username='slave'} at 127.2.182.2:46417; Asking this server to re-register.
I20250904 19:58:09.998487 3144 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:09.999092 3144 heartbeater.cc:507] Master 127.2.182.62:46421 requested a full tablet report, sending...
I20250904 19:58:10.001719 3502 ts_manager.cc:194] Registered new tserver with Master: daeded77000f4db39e0a8a9a764d0339 (127.2.182.2:46295)
I20250904 19:58:10.005646 3502 catalog_manager.cc:5582] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 reported cstate change: term changed from 1 to 2, leader changed from 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4) to 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3). New cstate: current_term: 2 leader_uuid: "6bd9b70c6b93412e992a48dceae23d6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } }
I20250904 19:58:10.014315 3501 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" instance_seqno: 1757015884036636) as {username='slave'} at 127.2.182.3:42333; Asking this server to re-register.
I20250904 19:58:10.015758 3277 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:10.016369 3277 heartbeater.cc:507] Master 127.2.182.62:46421 requested a full tablet report, sending...
I20250904 19:58:10.019269 3501 ts_manager.cc:194] Registered new tserver with Master: 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3:39175)
W20250904 19:58:10.704213 3539 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.366s user 0.525s sys 0.833s
W20250904 19:58:10.704578 3539 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.366s user 0.525s sys 0.833s
I20250904 19:58:10.776144 3099 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "daeded77000f4db39e0a8a9a764d0339"
tablet_id: "520b25df37424f83aef6dd893db557e7"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f"
}
peers {
permanent_uuid: "daeded77000f4db39e0a8a9a764d0339"
}
}
from {username='slave'} at 127.0.0.1:35510
W20250904 19:58:10.778040 3099 raft_consensus.cc:2216] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } unsafe_config_change: true
I20250904 19:58:10.779490 3099 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 2 FOLLOWER]: Advancing to term 3
I20250904 19:58:11.047878 3099 raft_consensus.cc:1238] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 3 FOLLOWER]: Rejecting Update request from peer 6bd9b70c6b93412e992a48dceae23d6f for earlier term 2. Current term is 3. Ops: []
I20250904 19:58:11.048870 3545 consensus_queue.cc:1046] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250904 19:58:11.050218 3596 raft_consensus.cc:3053] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 2 LEADER]: Stepping down as leader of term 2
I20250904 19:58:11.050446 3596 raft_consensus.cc:738] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bd9b70c6b93412e992a48dceae23d6f, State: Running, Role: LEADER
I20250904 19:58:11.051002 3596 consensus_queue.cc:260] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } }
I20250904 19:58:11.051847 3596 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 2 FOLLOWER]: Advancing to term 3
I20250904 19:58:12.289036 3599 raft_consensus.cc:491] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250904 19:58:12.289429 3599 raft_consensus.cc:513] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } unsafe_config_change: true
I20250904 19:58:12.290447 3599 leader_election.cc:290] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3:39175)
I20250904 19:58:12.291492 3232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "daeded77000f4db39e0a8a9a764d0339" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6bd9b70c6b93412e992a48dceae23d6f" is_pre_election: true
I20250904 19:58:12.291932 3232 raft_consensus.cc:2466] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate daeded77000f4db39e0a8a9a764d0339 in term 3.
I20250904 19:58:12.292842 3035 leader_election.cc:304] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6bd9b70c6b93412e992a48dceae23d6f, daeded77000f4db39e0a8a9a764d0339; no voters:
I20250904 19:58:12.293398 3599 raft_consensus.cc:2802] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250904 19:58:12.293644 3599 raft_consensus.cc:491] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250904 19:58:12.293849 3599 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 3 FOLLOWER]: Advancing to term 4
I20250904 19:58:12.297969 3599 raft_consensus.cc:513] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } unsafe_config_change: true
I20250904 19:58:12.298916 3599 leader_election.cc:290] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [CANDIDATE]: Term 4 election: Requested vote from peers 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3:39175)
I20250904 19:58:12.299844 3232 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "520b25df37424f83aef6dd893db557e7" candidate_uuid: "daeded77000f4db39e0a8a9a764d0339" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6bd9b70c6b93412e992a48dceae23d6f"
I20250904 19:58:12.300210 3232 raft_consensus.cc:3058] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 3 FOLLOWER]: Advancing to term 4
I20250904 19:58:12.304211 3232 raft_consensus.cc:2466] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daeded77000f4db39e0a8a9a764d0339 in term 4.
I20250904 19:58:12.304922 3035 leader_election.cc:304] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6bd9b70c6b93412e992a48dceae23d6f, daeded77000f4db39e0a8a9a764d0339; no voters:
I20250904 19:58:12.305472 3599 raft_consensus.cc:2802] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 4 FOLLOWER]: Leader election won for term 4
I20250904 19:58:12.306254 3599 raft_consensus.cc:695] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 4 LEADER]: Becoming Leader. State: Replica: daeded77000f4db39e0a8a9a764d0339, State: Running, Role: LEADER
I20250904 19:58:12.306912 3599 consensus_queue.cc:237] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } unsafe_config_change: true
I20250904 19:58:12.313107 3501 catalog_manager.cc:5582] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 reported cstate change: term changed from 2 to 4, leader changed from 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3) to daeded77000f4db39e0a8a9a764d0339 (127.2.182.2), now has a pending config: VOTER 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3), VOTER daeded77000f4db39e0a8a9a764d0339 (127.2.182.2). New cstate: current_term: 4 leader_uuid: "daeded77000f4db39e0a8a9a764d0339" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98ad5475edce46b6bbc61ecbdc5b5ad4" member_type: VOTER last_known_addr { host: "127.2.182.4" port: 46277 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } unsafe_config_change: true }
I20250904 19:58:12.711907 3232 raft_consensus.cc:1273] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 4 FOLLOWER]: Refusing update from remote peer daeded77000f4db39e0a8a9a764d0339: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250904 19:58:12.713332 3599 consensus_queue.cc:1035] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250904 19:58:12.721400 3600 raft_consensus.cc:2953] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } unsafe_config_change: true }
I20250904 19:58:12.722769 3232 raft_consensus.cc:2953] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } unsafe_config_change: true }
I20250904 19:58:12.733659 3501 catalog_manager.cc:5582] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 reported cstate change: config changed from index -1 to 3, VOTER 98ad5475edce46b6bbc61ecbdc5b5ad4 (127.2.182.4) evicted, no longer has a pending config: VOTER 6bd9b70c6b93412e992a48dceae23d6f (127.2.182.3), VOTER daeded77000f4db39e0a8a9a764d0339 (127.2.182.2). New cstate: current_term: 4 leader_uuid: "daeded77000f4db39e0a8a9a764d0339" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250904 19:58:12.740501 3501 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 520b25df37424f83aef6dd893db557e7 on TS 98ad5475edce46b6bbc61ecbdc5b5ad4: Not found: failed to reset TS proxy: Could not find TS for UUID 98ad5475edce46b6bbc61ecbdc5b5ad4
I20250904 19:58:12.754899 3099 consensus_queue.cc:237] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: true } } unsafe_config_change: true
I20250904 19:58:12.759234 3232 raft_consensus.cc:1273] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 4 FOLLOWER]: Refusing update from remote peer daeded77000f4db39e0a8a9a764d0339: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250904 19:58:12.760615 3600 consensus_queue.cc:1035] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250904 19:58:12.765527 3599 raft_consensus.cc:2953] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: true } } unsafe_config_change: true }
I20250904 19:58:12.767017 3232 raft_consensus.cc:2953] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: true } } unsafe_config_change: true }
W20250904 19:58:12.770442 3032 consensus_peers.cc:489] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 -> Peer cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1:46703): Couldn't send request to peer cfd8a6ead99048f5beac07b5ee5a2710. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 520b25df37424f83aef6dd893db557e7. This is attempt 1: this message will repeat every 5th retry.
I20250904 19:58:12.773448 3488 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 520b25df37424f83aef6dd893db557e7 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250904 19:58:12.776932 3502 catalog_manager.cc:5582] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 reported cstate change: config changed from index 3 to 5, NON_VOTER cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1) added. New cstate: current_term: 4 leader_uuid: "daeded77000f4db39e0a8a9a764d0339" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250904 19:58:12.792897 3487 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 520b25df37424f83aef6dd893db557e7 on TS 98ad5475edce46b6bbc61ecbdc5b5ad4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 98ad5475edce46b6bbc61ecbdc5b5ad4
I20250904 19:58:13.316148 3616 ts_tablet_manager.cc:927] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: Initiating tablet copy from peer daeded77000f4db39e0a8a9a764d0339 (127.2.182.2:46295)
I20250904 19:58:13.318168 3616 tablet_copy_client.cc:323] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.2:46295
I20250904 19:58:13.327432 3119 tablet_copy_service.cc:140] P daeded77000f4db39e0a8a9a764d0339: Received BeginTabletCopySession request for tablet 520b25df37424f83aef6dd893db557e7 from peer cfd8a6ead99048f5beac07b5ee5a2710 ({username='slave'} at 127.2.182.1:40843)
I20250904 19:58:13.327823 3119 tablet_copy_service.cc:161] P daeded77000f4db39e0a8a9a764d0339: Beginning new tablet copy session on tablet 520b25df37424f83aef6dd893db557e7 from peer cfd8a6ead99048f5beac07b5ee5a2710 at {username='slave'} at 127.2.182.1:40843: session id = cfd8a6ead99048f5beac07b5ee5a2710-520b25df37424f83aef6dd893db557e7
I20250904 19:58:13.332307 3119 tablet_copy_source_session.cc:215] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: Tablet Copy: opened 0 blocks and 1 log segments
I20250904 19:58:13.336584 3616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 520b25df37424f83aef6dd893db557e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:13.354562 3616 tablet_copy_client.cc:806] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: tablet copy: Starting download of 0 data blocks...
I20250904 19:58:13.355064 3616 tablet_copy_client.cc:670] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: tablet copy: Starting download of 1 WAL segments...
I20250904 19:58:13.358001 3616 tablet_copy_client.cc:538] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250904 19:58:13.363108 3616 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: Bootstrap starting.
I20250904 19:58:13.373968 3616 log.cc:826] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:13.387497 3616 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:58:13.388320 3616 tablet_bootstrap.cc:492] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: Bootstrap complete.
I20250904 19:58:13.388993 3616 ts_tablet_manager.cc:1397] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: Time spent bootstrapping tablet: real 0.026s user 0.022s sys 0.001s
I20250904 19:58:13.407138 3616 raft_consensus.cc:357] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: true } } unsafe_config_change: true
I20250904 19:58:13.408272 3616 raft_consensus.cc:738] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: cfd8a6ead99048f5beac07b5ee5a2710, State: Initialized, Role: LEARNER
I20250904 19:58:13.408895 3616 consensus_queue.cc:260] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: true } } unsafe_config_change: true
I20250904 19:58:13.413281 3616 ts_tablet_manager.cc:1428] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710: Time spent starting tablet: real 0.024s user 0.021s sys 0.004s
I20250904 19:58:13.415154 3119 tablet_copy_service.cc:342] P daeded77000f4db39e0a8a9a764d0339: Request end of tablet copy session cfd8a6ead99048f5beac07b5ee5a2710-520b25df37424f83aef6dd893db557e7 received from {username='slave'} at 127.2.182.1:40843
I20250904 19:58:13.415602 3119 tablet_copy_service.cc:434] P daeded77000f4db39e0a8a9a764d0339: ending tablet copy session cfd8a6ead99048f5beac07b5ee5a2710-520b25df37424f83aef6dd893db557e7 on tablet 520b25df37424f83aef6dd893db557e7 with peer cfd8a6ead99048f5beac07b5ee5a2710
I20250904 19:58:13.828255 2966 raft_consensus.cc:1215] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250904 19:58:13.959522 3487 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 520b25df37424f83aef6dd893db557e7 on TS 98ad5475edce46b6bbc61ecbdc5b5ad4 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 98ad5475edce46b6bbc61ecbdc5b5ad4
I20250904 19:58:14.359826 3623 raft_consensus.cc:1062] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339: attempting to promote NON_VOTER cfd8a6ead99048f5beac07b5ee5a2710 to VOTER
I20250904 19:58:14.361343 3623 consensus_queue.cc:237] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: false } } unsafe_config_change: true
I20250904 19:58:14.365916 2966 raft_consensus.cc:1273] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710 [term 4 LEARNER]: Refusing update from remote peer daeded77000f4db39e0a8a9a764d0339: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250904 19:58:14.367112 3623 consensus_queue.cc:1035] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 19:58:14.367376 3232 raft_consensus.cc:1273] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 4 FOLLOWER]: Refusing update from remote peer daeded77000f4db39e0a8a9a764d0339: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250904 19:58:14.368599 3624 consensus_queue.cc:1035] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250904 19:58:14.374367 3623 raft_consensus.cc:2953] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: false } } unsafe_config_change: true }
I20250904 19:58:14.376078 2966 raft_consensus.cc:2953] T 520b25df37424f83aef6dd893db557e7 P cfd8a6ead99048f5beac07b5ee5a2710 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: false } } unsafe_config_change: true }
I20250904 19:58:14.378067 3232 raft_consensus.cc:2953] T 520b25df37424f83aef6dd893db557e7 P 6bd9b70c6b93412e992a48dceae23d6f [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: false } } unsafe_config_change: true }
I20250904 19:58:14.390167 3502 catalog_manager.cc:5582] T 520b25df37424f83aef6dd893db557e7 P daeded77000f4db39e0a8a9a764d0339 reported cstate change: config changed from index 5 to 6, cfd8a6ead99048f5beac07b5ee5a2710 (127.2.182.1) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "daeded77000f4db39e0a8a9a764d0339" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daeded77000f4db39e0a8a9a764d0339" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 46295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 46703 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250904 19:58:14.459151 2776 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250904 19:58:14.470311 2776 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "520b25df37424f83aef6dd893db557e7"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "6bd9b70c6b93412e992a48dceae23d6f"
rpc_addresses {
host: "127.2.182.3"
port: 39175
}
}
ts_infos {
permanent_uuid: "daeded77000f4db39e0a8a9a764d0339"
rpc_addresses {
host: "127.2.182.2"
port: 46295
}
}
ts_infos {
permanent_uuid: "cfd8a6ead99048f5beac07b5ee5a2710"
rpc_addresses {
host: "127.2.182.1"
port: 46703
}
}
I20250904 19:58:14.473492 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 2882
I20250904 19:58:14.493347 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 3015
I20250904 19:58:14.513161 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 3148
I20250904 19:58:14.532096 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 3455
2025-09-04T19:58:14Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19130 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250904 19:58:14.585129 2776 test_util.cc:276] Using random seed: 608366354
I20250904 19:58:14.590662 2776 ts_itest-base.cc:115] Starting cluster with:
I20250904 19:58:14.590847 2776 ts_itest-base.cc:116] --------------
I20250904 19:58:14.591005 2776 ts_itest-base.cc:117] 3 tablet servers
I20250904 19:58:14.591140 2776 ts_itest-base.cc:118] 3 replicas per TS
I20250904 19:58:14.591281 2776 ts_itest-base.cc:119] --------------
2025-09-04T19:58:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T19:58:14Z Disabled control of system clock
I20250904 19:58:14.622036 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:35921
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:35921
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250904 19:58:14.901206 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:14.901695 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:14.902092 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:14.931658 3645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:58:14.931918 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:14.932116 3645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:58:14.932304 3645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:58:14.965730 3645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:35921
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:14.966924 3645 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:14.968443 3645 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:14.977895 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:14.978974 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:16.161315 3653 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1177 milliseconds
W20250904 19:58:16.161952 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:16.163959 3645 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.186s user 0.005s sys 0.003s
W20250904 19:58:16.164227 3645 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.186s user 0.005s sys 0.003s
I20250904 19:58:16.164438 3645 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:16.165450 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:16.168145 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:16.169541 3645 hybrid_clock.cc:648] HybridClock initialized: now 1757015896169487 us; error 61 us; skew 500 ppm
I20250904 19:58:16.170334 3645 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:16.177764 3645 webserver.cc:480] Webserver started at http://127.2.182.62:44619/ using document root <none> and password file <none>
I20250904 19:58:16.178980 3645 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:16.179183 3645 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:16.179589 3645 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:16.183784 3645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d1346f0989634dc085879f767eab42f4"
format_stamp: "Formatted at 2025-09-04 19:58:16 on dist-test-slave-tjsx"
I20250904 19:58:16.184859 3645 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d1346f0989634dc085879f767eab42f4"
format_stamp: "Formatted at 2025-09-04 19:58:16 on dist-test-slave-tjsx"
I20250904 19:58:16.192286 3645 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250904 19:58:16.197968 3662 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:16.198989 3645 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250904 19:58:16.199293 3645 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d1346f0989634dc085879f767eab42f4"
format_stamp: "Formatted at 2025-09-04 19:58:16 on dist-test-slave-tjsx"
I20250904 19:58:16.199589 3645 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:16.259243 3645 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:16.260655 3645 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:16.261070 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:16.325455 3645 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:35921
I20250904 19:58:16.325556 3713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:35921 every 8 connection(s)
I20250904 19:58:16.328011 3645 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:58:16.332619 3714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:16.336191 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 3645
I20250904 19:58:16.336763 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250904 19:58:16.353996 3714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4: Bootstrap starting.
I20250904 19:58:16.359457 3714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:16.361501 3714 log.cc:826] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:16.365625 3714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4: No bootstrap required, opened a new log
I20250904 19:58:16.382349 3714 raft_consensus.cc:357] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1346f0989634dc085879f767eab42f4" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35921 } }
I20250904 19:58:16.382992 3714 raft_consensus.cc:383] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:16.383224 3714 raft_consensus.cc:738] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1346f0989634dc085879f767eab42f4, State: Initialized, Role: FOLLOWER
I20250904 19:58:16.383880 3714 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1346f0989634dc085879f767eab42f4" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35921 } }
I20250904 19:58:16.384387 3714 raft_consensus.cc:397] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:58:16.384646 3714 raft_consensus.cc:491] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:58:16.385010 3714 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:16.389024 3714 raft_consensus.cc:513] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1346f0989634dc085879f767eab42f4" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35921 } }
I20250904 19:58:16.389652 3714 leader_election.cc:304] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1346f0989634dc085879f767eab42f4; no voters:
I20250904 19:58:16.391196 3714 leader_election.cc:290] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:58:16.391865 3719 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:16.393791 3719 raft_consensus.cc:695] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [term 1 LEADER]: Becoming Leader. State: Replica: d1346f0989634dc085879f767eab42f4, State: Running, Role: LEADER
I20250904 19:58:16.395092 3719 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1346f0989634dc085879f767eab42f4" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35921 } }
I20250904 19:58:16.395633 3714 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:58:16.401552 3720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1346f0989634dc085879f767eab42f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1346f0989634dc085879f767eab42f4" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35921 } } }
I20250904 19:58:16.402244 3720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:16.402462 3721 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1346f0989634dc085879f767eab42f4. Latest consensus state: current_term: 1 leader_uuid: "d1346f0989634dc085879f767eab42f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1346f0989634dc085879f767eab42f4" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 35921 } } }
I20250904 19:58:16.403034 3721 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4 [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:16.405843 3726 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:58:16.417126 3726 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:58:16.430454 3726 catalog_manager.cc:1349] Generated new cluster ID: 2ee4098b14e34895a31fe91ca96bb284
I20250904 19:58:16.430760 3726 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:58:16.445313 3726 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 19:58:16.447175 3726 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:58:16.463662 3726 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d1346f0989634dc085879f767eab42f4: Generated new TSK 0
I20250904 19:58:16.464591 3726 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 19:58:16.475803 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35921
--builtin_ntp_servers=127.2.182.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250904 19:58:16.758915 3738 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250904 19:58:16.759501 3738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:16.759742 3738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:16.760210 3738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:16.790947 3738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:16.791769 3738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 19:58:16.826131 3738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:16.827394 3738 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:16.828927 3738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:16.840633 3744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:16.842379 3745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:17.945043 3747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:17.946965 3746 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1101 milliseconds
I20250904 19:58:17.947134 3738 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:17.948607 3738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:17.951305 3738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:17.952800 3738 hybrid_clock.cc:648] HybridClock initialized: now 1757015897952749 us; error 38 us; skew 500 ppm
I20250904 19:58:17.953920 3738 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:17.961313 3738 webserver.cc:480] Webserver started at http://127.2.182.1:33783/ using document root <none> and password file <none>
I20250904 19:58:17.962560 3738 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:17.962854 3738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:17.963507 3738 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:17.969712 3738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7f12c9155fb344368390b5eaa607543a"
format_stamp: "Formatted at 2025-09-04 19:58:17 on dist-test-slave-tjsx"
I20250904 19:58:17.971201 3738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7f12c9155fb344368390b5eaa607543a"
format_stamp: "Formatted at 2025-09-04 19:58:17 on dist-test-slave-tjsx"
I20250904 19:58:17.979954 3738 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250904 19:58:17.986922 3754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:17.987975 3738 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250904 19:58:17.988384 3738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7f12c9155fb344368390b5eaa607543a"
format_stamp: "Formatted at 2025-09-04 19:58:17 on dist-test-slave-tjsx"
I20250904 19:58:17.988781 3738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:18.048514 3738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:18.049875 3738 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:18.050289 3738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:18.052814 3738 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:18.056743 3738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:18.056946 3738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:18.057183 3738 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:18.057333 3738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:18.203007 3738 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:35521
I20250904 19:58:18.203166 3866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:35521 every 8 connection(s)
I20250904 19:58:18.205464 3738 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 19:58:18.215879 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 3738
I20250904 19:58:18.216308 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250904 19:58:18.223129 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35921
--builtin_ntp_servers=127.2.182.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250904 19:58:18.226135 3867 heartbeater.cc:344] Connected to a master server at 127.2.182.62:35921
I20250904 19:58:18.226583 3867 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:18.227813 3867 heartbeater.cc:507] Master 127.2.182.62:35921 requested a full tablet report, sending...
I20250904 19:58:18.230743 3679 ts_manager.cc:194] Registered new tserver with Master: 7f12c9155fb344368390b5eaa607543a (127.2.182.1:35521)
I20250904 19:58:18.233402 3679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:56369
W20250904 19:58:18.526335 3871 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250904 19:58:18.526959 3871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:18.527220 3871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:18.527680 3871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:18.558251 3871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:18.559101 3871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 19:58:18.593386 3871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:18.594779 3871 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:18.596421 3871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:18.607596 3877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:19.237244 3867 heartbeater.cc:499] Master 127.2.182.62:35921 was elected leader, sending a full tablet report...
W20250904 19:58:18.608204 3878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:19.785493 3880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:19.788503 3879 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1179 milliseconds
W20250904 19:58:19.789034 3871 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.182s user 0.478s sys 0.699s
W20250904 19:58:19.789273 3871 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.182s user 0.478s sys 0.699s
I20250904 19:58:19.789462 3871 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:19.790452 3871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:19.792567 3871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:19.793913 3871 hybrid_clock.cc:648] HybridClock initialized: now 1757015899793884 us; error 38 us; skew 500 ppm
I20250904 19:58:19.794657 3871 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:19.801717 3871 webserver.cc:480] Webserver started at http://127.2.182.2:44855/ using document root <none> and password file <none>
I20250904 19:58:19.802604 3871 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:19.802841 3871 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:19.803308 3871 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:19.808265 3871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "107a83c4972f46699a9ea630cf2bed27"
format_stamp: "Formatted at 2025-09-04 19:58:19 on dist-test-slave-tjsx"
I20250904 19:58:19.809725 3871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "107a83c4972f46699a9ea630cf2bed27"
format_stamp: "Formatted at 2025-09-04 19:58:19 on dist-test-slave-tjsx"
I20250904 19:58:19.817919 3871 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250904 19:58:19.823508 3887 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:19.824555 3871 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250904 19:58:19.824870 3871 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "107a83c4972f46699a9ea630cf2bed27"
format_stamp: "Formatted at 2025-09-04 19:58:19 on dist-test-slave-tjsx"
I20250904 19:58:19.825178 3871 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:19.881914 3871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:19.883447 3871 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:19.883934 3871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:19.886288 3871 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:19.890066 3871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:19.890272 3871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:19.890498 3871 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:19.890653 3871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:20.028272 3871 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:42911
I20250904 19:58:20.028357 3999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:42911 every 8 connection(s)
I20250904 19:58:20.030761 3871 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250904 19:58:20.037415 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 3871
I20250904 19:58:20.037745 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250904 19:58:20.043104 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35921
--builtin_ntp_servers=127.2.182.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250904 19:58:20.049412 4000 heartbeater.cc:344] Connected to a master server at 127.2.182.62:35921
I20250904 19:58:20.049866 4000 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:20.050849 4000 heartbeater.cc:507] Master 127.2.182.62:35921 requested a full tablet report, sending...
I20250904 19:58:20.052874 3679 ts_manager.cc:194] Registered new tserver with Master: 107a83c4972f46699a9ea630cf2bed27 (127.2.182.2:42911)
I20250904 19:58:20.054456 3679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:38709
W20250904 19:58:20.327975 4004 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250904 19:58:20.328541 4004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:20.328857 4004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:20.329332 4004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:20.359688 4004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:20.360492 4004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 19:58:20.394593 4004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:20.395816 4004 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:20.397320 4004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:20.408321 4010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:21.057300 4000 heartbeater.cc:499] Master 127.2.182.62:35921 was elected leader, sending a full tablet report...
W20250904 19:58:20.408942 4011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:21.473800 4012 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250904 19:58:21.475155 4013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:21.475196 4004 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:21.477624 4004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:21.479849 4004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:21.481174 4004 hybrid_clock.cc:648] HybridClock initialized: now 1757015901481136 us; error 40 us; skew 500 ppm
I20250904 19:58:21.481920 4004 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:21.487721 4004 webserver.cc:480] Webserver started at http://127.2.182.3:38287/ using document root <none> and password file <none>
I20250904 19:58:21.488574 4004 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:21.488767 4004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:21.489202 4004 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:21.493690 4004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "69e75881cad848c090188005fb6e9945"
format_stamp: "Formatted at 2025-09-04 19:58:21 on dist-test-slave-tjsx"
I20250904 19:58:21.494737 4004 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "69e75881cad848c090188005fb6e9945"
format_stamp: "Formatted at 2025-09-04 19:58:21 on dist-test-slave-tjsx"
I20250904 19:58:21.501538 4004 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250904 19:58:21.506613 4020 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:21.507542 4004 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 19:58:21.507860 4004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "69e75881cad848c090188005fb6e9945"
format_stamp: "Formatted at 2025-09-04 19:58:21 on dist-test-slave-tjsx"
I20250904 19:58:21.508158 4004 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:21.564486 4004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:21.565887 4004 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:21.566279 4004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:21.568608 4004 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:21.572535 4004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:21.572721 4004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:21.572969 4004 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:21.573117 4004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:21.695508 4004 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:44213
I20250904 19:58:21.695598 4132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:44213 every 8 connection(s)
I20250904 19:58:21.697930 4004 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250904 19:58:21.702025 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 4004
I20250904 19:58:21.702448 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250904 19:58:21.720834 4133 heartbeater.cc:344] Connected to a master server at 127.2.182.62:35921
I20250904 19:58:21.721328 4133 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:21.722401 4133 heartbeater.cc:507] Master 127.2.182.62:35921 requested a full tablet report, sending...
I20250904 19:58:21.724296 3679 ts_manager.cc:194] Registered new tserver with Master: 69e75881cad848c090188005fb6e9945 (127.2.182.3:44213)
I20250904 19:58:21.725430 3679 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:46995
I20250904 19:58:21.735224 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:58:21.767508 3679 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33766:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250904 19:58:21.785326 3679 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 19:58:21.836557 3802 tablet_service.cc:1468] Processing CreateTablet for tablet 4651f84f88154834a7b17ec552a4f67b (DEFAULT_TABLE table=TestTable [id=c9d0a9f364d84574b393f8b5ddef07c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:21.838223 3802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4651f84f88154834a7b17ec552a4f67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:21.842849 3935 tablet_service.cc:1468] Processing CreateTablet for tablet 4651f84f88154834a7b17ec552a4f67b (DEFAULT_TABLE table=TestTable [id=c9d0a9f364d84574b393f8b5ddef07c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:21.844657 3935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4651f84f88154834a7b17ec552a4f67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:21.844648 4068 tablet_service.cc:1468] Processing CreateTablet for tablet 4651f84f88154834a7b17ec552a4f67b (DEFAULT_TABLE table=TestTable [id=c9d0a9f364d84574b393f8b5ddef07c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:21.846426 4068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4651f84f88154834a7b17ec552a4f67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:21.864053 4152 tablet_bootstrap.cc:492] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27: Bootstrap starting.
I20250904 19:58:21.865406 4153 tablet_bootstrap.cc:492] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a: Bootstrap starting.
I20250904 19:58:21.870815 4154 tablet_bootstrap.cc:492] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945: Bootstrap starting.
I20250904 19:58:21.872087 4152 tablet_bootstrap.cc:654] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:21.873574 4153 tablet_bootstrap.cc:654] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:21.875135 4152 log.cc:826] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:21.875762 4153 log.cc:826] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:21.876646 4154 tablet_bootstrap.cc:654] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:21.878965 4154 log.cc:826] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:21.880079 4152 tablet_bootstrap.cc:492] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27: No bootstrap required, opened a new log
I20250904 19:58:21.880488 4152 ts_tablet_manager.cc:1397] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20250904 19:58:21.881009 4153 tablet_bootstrap.cc:492] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a: No bootstrap required, opened a new log
I20250904 19:58:21.881501 4153 ts_tablet_manager.cc:1397] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20250904 19:58:21.884395 4154 tablet_bootstrap.cc:492] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945: No bootstrap required, opened a new log
I20250904 19:58:21.884876 4154 ts_tablet_manager.cc:1397] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20250904 19:58:21.900445 4153 raft_consensus.cc:357] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:21.901149 4153 raft_consensus.cc:738] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f12c9155fb344368390b5eaa607543a, State: Initialized, Role: FOLLOWER
I20250904 19:58:21.901844 4153 consensus_queue.cc:260] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:21.906071 4152 raft_consensus.cc:357] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:21.906985 4152 raft_consensus.cc:738] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 107a83c4972f46699a9ea630cf2bed27, State: Initialized, Role: FOLLOWER
I20250904 19:58:21.907693 4152 consensus_queue.cc:260] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:21.908644 4153 ts_tablet_manager.cc:1428] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a: Time spent starting tablet: real 0.027s user 0.026s sys 0.001s
I20250904 19:58:21.910905 4154 raft_consensus.cc:357] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:21.912005 4152 ts_tablet_manager.cc:1428] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27: Time spent starting tablet: real 0.031s user 0.026s sys 0.004s
I20250904 19:58:21.911867 4154 raft_consensus.cc:738] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e75881cad848c090188005fb6e9945, State: Initialized, Role: FOLLOWER
I20250904 19:58:21.912580 4154 consensus_queue.cc:260] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:21.915367 4133 heartbeater.cc:499] Master 127.2.182.62:35921 was elected leader, sending a full tablet report...
I20250904 19:58:21.916460 4154 ts_tablet_manager.cc:1428] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945: Time spent starting tablet: real 0.031s user 0.025s sys 0.006s
I20250904 19:58:21.935245 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:58:21.938256 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7f12c9155fb344368390b5eaa607543a to finish bootstrapping
I20250904 19:58:21.950436 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 107a83c4972f46699a9ea630cf2bed27 to finish bootstrapping
W20250904 19:58:21.952023 4134 tablet.cc:2378] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:21.960412 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 69e75881cad848c090188005fb6e9945 to finish bootstrapping
W20250904 19:58:21.963011 3868 tablet.cc:2378] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:21.998414 3822 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4651f84f88154834a7b17ec552a4f67b"
dest_uuid: "7f12c9155fb344368390b5eaa607543a"
from {username='slave'} at 127.0.0.1:53426
I20250904 19:58:21.998982 3822 raft_consensus.cc:491] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250904 19:58:21.999289 3822 raft_consensus.cc:3058] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:22.003528 3822 raft_consensus.cc:513] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:22.005723 3822 leader_election.cc:290] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [CANDIDATE]: Term 1 election: Requested vote from peers 107a83c4972f46699a9ea630cf2bed27 (127.2.182.2:42911), 69e75881cad848c090188005fb6e9945 (127.2.182.3:44213)
I20250904 19:58:22.014811 2776 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250904 19:58:22.017150 3955 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4651f84f88154834a7b17ec552a4f67b" candidate_uuid: "7f12c9155fb344368390b5eaa607543a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "107a83c4972f46699a9ea630cf2bed27"
I20250904 19:58:22.017746 3955 raft_consensus.cc:3058] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:22.017608 4088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4651f84f88154834a7b17ec552a4f67b" candidate_uuid: "7f12c9155fb344368390b5eaa607543a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "69e75881cad848c090188005fb6e9945"
I20250904 19:58:22.018124 4088 raft_consensus.cc:3058] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:22.021888 3955 raft_consensus.cc:2466] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f12c9155fb344368390b5eaa607543a in term 1.
I20250904 19:58:22.022260 4088 raft_consensus.cc:2466] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f12c9155fb344368390b5eaa607543a in term 1.
I20250904 19:58:22.022862 3758 leader_election.cc:304] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107a83c4972f46699a9ea630cf2bed27, 7f12c9155fb344368390b5eaa607543a; no voters:
I20250904 19:58:22.023669 4158 raft_consensus.cc:2802] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:22.025154 4158 raft_consensus.cc:695] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 1 LEADER]: Becoming Leader. State: Replica: 7f12c9155fb344368390b5eaa607543a, State: Running, Role: LEADER
I20250904 19:58:22.025949 4158 consensus_queue.cc:237] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
W20250904 19:58:22.036489 4001 tablet.cc:2378] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:22.038460 3677 catalog_manager.cc:5582] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f12c9155fb344368390b5eaa607543a (127.2.182.1). New cstate: current_term: 1 leader_uuid: "7f12c9155fb344368390b5eaa607543a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } health_report { overall_health: UNKNOWN } } }
I20250904 19:58:22.119850 2776 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250904 19:58:22.325066 2776 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250904 19:58:22.500229 4158 consensus_queue.cc:1035] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 19:58:22.514726 4158 consensus_queue.cc:1035] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [LEADER]: Connected to new peer: Peer: permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 19:58:24.149650 3822 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4651f84f88154834a7b17ec552a4f67b"
dest_uuid: "7f12c9155fb344368390b5eaa607543a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42412
I20250904 19:58:24.150122 3822 raft_consensus.cc:604] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 1 LEADER]: Received request to transfer leadership
I20250904 19:58:24.540846 4185 raft_consensus.cc:991] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a: : Instructing follower 107a83c4972f46699a9ea630cf2bed27 to start an election
I20250904 19:58:24.541181 4185 raft_consensus.cc:1079] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 1 LEADER]: Signalling peer 107a83c4972f46699a9ea630cf2bed27 to start an election
I20250904 19:58:24.542373 3955 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4651f84f88154834a7b17ec552a4f67b"
dest_uuid: "107a83c4972f46699a9ea630cf2bed27"
from {username='slave'} at 127.2.182.1:44437
I20250904 19:58:24.542891 3955 raft_consensus.cc:491] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250904 19:58:24.543210 3955 raft_consensus.cc:3058] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 1 FOLLOWER]: Advancing to term 2
I20250904 19:58:24.547322 3955 raft_consensus.cc:513] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:24.549414 3955 leader_election.cc:290] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [CANDIDATE]: Term 2 election: Requested vote from peers 7f12c9155fb344368390b5eaa607543a (127.2.182.1:35521), 69e75881cad848c090188005fb6e9945 (127.2.182.3:44213)
I20250904 19:58:24.559580 3822 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4651f84f88154834a7b17ec552a4f67b" candidate_uuid: "107a83c4972f46699a9ea630cf2bed27" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7f12c9155fb344368390b5eaa607543a"
I20250904 19:58:24.560108 3822 raft_consensus.cc:3053] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 1 LEADER]: Stepping down as leader of term 1
I20250904 19:58:24.560341 3822 raft_consensus.cc:738] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f12c9155fb344368390b5eaa607543a, State: Running, Role: LEADER
I20250904 19:58:24.560845 4088 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4651f84f88154834a7b17ec552a4f67b" candidate_uuid: "107a83c4972f46699a9ea630cf2bed27" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "69e75881cad848c090188005fb6e9945"
I20250904 19:58:24.560941 3822 consensus_queue.cc:260] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:24.561395 4088 raft_consensus.cc:3058] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945 [term 1 FOLLOWER]: Advancing to term 2
I20250904 19:58:24.562181 3822 raft_consensus.cc:3058] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 1 FOLLOWER]: Advancing to term 2
I20250904 19:58:24.567772 4088 raft_consensus.cc:2466] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 107a83c4972f46699a9ea630cf2bed27 in term 2.
I20250904 19:58:24.568437 3822 raft_consensus.cc:2466] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 107a83c4972f46699a9ea630cf2bed27 in term 2.
I20250904 19:58:24.568998 3889 leader_election.cc:304] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107a83c4972f46699a9ea630cf2bed27, 69e75881cad848c090188005fb6e9945; no voters:
I20250904 19:58:24.570842 4199 raft_consensus.cc:2802] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 19:58:24.572009 4199 raft_consensus.cc:695] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [term 2 LEADER]: Becoming Leader. State: Replica: 107a83c4972f46699a9ea630cf2bed27, State: Running, Role: LEADER
I20250904 19:58:24.572726 4199 consensus_queue.cc:237] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } }
I20250904 19:58:24.579802 3677 catalog_manager.cc:5582] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 reported cstate change: term changed from 1 to 2, leader changed from 7f12c9155fb344368390b5eaa607543a (127.2.182.1) to 107a83c4972f46699a9ea630cf2bed27 (127.2.182.2). New cstate: current_term: 2 leader_uuid: "107a83c4972f46699a9ea630cf2bed27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107a83c4972f46699a9ea630cf2bed27" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 42911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 } health_report { overall_health: UNKNOWN } } }
I20250904 19:58:25.143405 4088 raft_consensus.cc:1273] T 4651f84f88154834a7b17ec552a4f67b P 69e75881cad848c090188005fb6e9945 [term 2 FOLLOWER]: Refusing update from remote peer 107a83c4972f46699a9ea630cf2bed27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250904 19:58:25.144615 4207 consensus_queue.cc:1035] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e75881cad848c090188005fb6e9945" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 44213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250904 19:58:25.155045 3822 raft_consensus.cc:1273] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 2 FOLLOWER]: Refusing update from remote peer 107a83c4972f46699a9ea630cf2bed27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250904 19:58:25.156652 4212 consensus_queue.cc:1035] T 4651f84f88154834a7b17ec552a4f67b P 107a83c4972f46699a9ea630cf2bed27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f12c9155fb344368390b5eaa607543a" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250904 19:58:26.734305 3822 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4651f84f88154834a7b17ec552a4f67b"
dest_uuid: "7f12c9155fb344368390b5eaa607543a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42428
I20250904 19:58:26.734774 3822 raft_consensus.cc:604] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 2 FOLLOWER]: Received request to transfer leadership
I20250904 19:58:26.735028 3822 raft_consensus.cc:612] T 4651f84f88154834a7b17ec552a4f67b P 7f12c9155fb344368390b5eaa607543a [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250904 19:58:27.773005 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 3738
I20250904 19:58:27.794979 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 3871
I20250904 19:58:27.822422 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 4004
I20250904 19:58:27.844578 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 3645
2025-09-04T19:58:27Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13310 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250904 19:58:27.896966 2776 test_util.cc:276] Using random seed: 621678176
I20250904 19:58:27.901887 2776 ts_itest-base.cc:115] Starting cluster with:
I20250904 19:58:27.902072 2776 ts_itest-base.cc:116] --------------
I20250904 19:58:27.902216 2776 ts_itest-base.cc:117] 3 tablet servers
I20250904 19:58:27.902333 2776 ts_itest-base.cc:118] 3 replicas per TS
I20250904 19:58:27.902454 2776 ts_itest-base.cc:119] --------------
2025-09-04T19:58:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T19:58:27Z Disabled control of system clock
I20250904 19:58:27.938796 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36531
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:36531 with env {}
W20250904 19:58:28.241565 4247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:28.242137 4247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:28.242545 4247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:28.274948 4247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:58:28.275298 4247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:28.275516 4247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:58:28.275868 4247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:58:28.314986 4247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:36531
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:28.316228 4247 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:28.317806 4247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:28.328433 4253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:28.329119 4254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:29.439203 4256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:29.442402 4255 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1107 milliseconds
I20250904 19:58:29.442569 4247 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:29.444092 4247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:29.447469 4247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:29.449020 4247 hybrid_clock.cc:648] HybridClock initialized: now 1757015909448923 us; error 103 us; skew 500 ppm
I20250904 19:58:29.449941 4247 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:29.456513 4247 webserver.cc:480] Webserver started at http://127.2.182.62:46231/ using document root <none> and password file <none>
I20250904 19:58:29.457461 4247 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:29.457691 4247 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:29.458153 4247 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:29.462555 4247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "36c5f2a30b7e4979818958bf9334291c"
format_stamp: "Formatted at 2025-09-04 19:58:29 on dist-test-slave-tjsx"
I20250904 19:58:29.463718 4247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "36c5f2a30b7e4979818958bf9334291c"
format_stamp: "Formatted at 2025-09-04 19:58:29 on dist-test-slave-tjsx"
I20250904 19:58:29.470841 4247 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250904 19:58:29.476329 4263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:29.477381 4247 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250904 19:58:29.477720 4247 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "36c5f2a30b7e4979818958bf9334291c"
format_stamp: "Formatted at 2025-09-04 19:58:29 on dist-test-slave-tjsx"
I20250904 19:58:29.478068 4247 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:29.536273 4247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:29.537784 4247 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:29.538223 4247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:29.606763 4247 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:36531
I20250904 19:58:29.606889 4314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:36531 every 8 connection(s)
I20250904 19:58:29.609414 4247 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:58:29.615099 4315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:29.617241 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 4247
I20250904 19:58:29.617735 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250904 19:58:29.639029 4315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c: Bootstrap starting.
I20250904 19:58:29.644884 4315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:29.646610 4315 log.cc:826] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:29.651315 4315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c: No bootstrap required, opened a new log
I20250904 19:58:29.668471 4315 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c5f2a30b7e4979818958bf9334291c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36531 } }
I20250904 19:58:29.669175 4315 raft_consensus.cc:383] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:29.669404 4315 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c5f2a30b7e4979818958bf9334291c, State: Initialized, Role: FOLLOWER
I20250904 19:58:29.670125 4315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c5f2a30b7e4979818958bf9334291c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36531 } }
I20250904 19:58:29.670656 4315 raft_consensus.cc:397] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:58:29.670989 4315 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:58:29.671293 4315 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:29.675527 4315 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c5f2a30b7e4979818958bf9334291c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36531 } }
I20250904 19:58:29.676276 4315 leader_election.cc:304] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36c5f2a30b7e4979818958bf9334291c; no voters:
I20250904 19:58:29.677955 4315 leader_election.cc:290] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:58:29.678684 4320 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:29.680733 4320 raft_consensus.cc:695] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [term 1 LEADER]: Becoming Leader. State: Replica: 36c5f2a30b7e4979818958bf9334291c, State: Running, Role: LEADER
I20250904 19:58:29.681596 4320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c5f2a30b7e4979818958bf9334291c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36531 } }
I20250904 19:58:29.682654 4315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:58:29.691249 4321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36c5f2a30b7e4979818958bf9334291c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c5f2a30b7e4979818958bf9334291c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36531 } } }
I20250904 19:58:29.691934 4321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:29.693141 4322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c5f2a30b7e4979818958bf9334291c. Latest consensus state: current_term: 1 leader_uuid: "36c5f2a30b7e4979818958bf9334291c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c5f2a30b7e4979818958bf9334291c" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36531 } } }
I20250904 19:58:29.693780 4322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:29.696434 4328 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:58:29.710021 4328 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:58:29.727335 4328 catalog_manager.cc:1349] Generated new cluster ID: 99b8f11f85cc4805b815a47acf954ee8
I20250904 19:58:29.727692 4328 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:58:29.762967 4328 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 19:58:29.764484 4328 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:58:29.783232 4328 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 36c5f2a30b7e4979818958bf9334291c: Generated new TSK 0
I20250904 19:58:29.784317 4328 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 19:58:29.807843 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36531
--builtin_ntp_servers=127.2.182.20:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250904 19:58:30.164808 4339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:30.165421 4339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:30.165964 4339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:30.200168 4339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:30.201121 4339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 19:58:30.246539 4339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:30.248268 4339 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:30.250268 4339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:30.264505 4345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:30.266486 4346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:31.625795 4348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:31.629626 4347 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1363 milliseconds
W20250904 19:58:31.630905 4339 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.366s user 0.454s sys 0.900s
W20250904 19:58:31.631282 4339 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.366s user 0.454s sys 0.901s
I20250904 19:58:31.631587 4339 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:31.633198 4339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:31.643260 4339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:31.644785 4339 hybrid_clock.cc:648] HybridClock initialized: now 1757015911644702 us; error 82 us; skew 500 ppm
I20250904 19:58:31.645972 4339 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:31.655494 4339 webserver.cc:480] Webserver started at http://127.2.182.1:41959/ using document root <none> and password file <none>
I20250904 19:58:31.656728 4339 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:31.657002 4339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:31.657554 4339 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:31.664186 4339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "92dbb6eab71a4095a7f207ce564314a3"
format_stamp: "Formatted at 2025-09-04 19:58:31 on dist-test-slave-tjsx"
I20250904 19:58:31.665714 4339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "92dbb6eab71a4095a7f207ce564314a3"
format_stamp: "Formatted at 2025-09-04 19:58:31 on dist-test-slave-tjsx"
I20250904 19:58:31.674935 4339 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.005s
I20250904 19:58:31.682576 4355 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:31.683905 4339 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250904 19:58:31.684345 4339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "92dbb6eab71a4095a7f207ce564314a3"
format_stamp: "Formatted at 2025-09-04 19:58:31 on dist-test-slave-tjsx"
I20250904 19:58:31.684762 4339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:31.762732 4339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:31.764201 4339 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:31.764631 4339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:31.767465 4339 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:31.771564 4339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:31.771759 4339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:31.772019 4339 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:31.772166 4339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:31.927947 4339 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:43243
I20250904 19:58:31.928047 4467 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:43243 every 8 connection(s)
I20250904 19:58:31.930512 4339 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 19:58:31.940454 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 4339
I20250904 19:58:31.940954 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250904 19:58:31.947834 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36531
--builtin_ntp_servers=127.2.182.20:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:58:31.958729 4468 heartbeater.cc:344] Connected to a master server at 127.2.182.62:36531
I20250904 19:58:31.959241 4468 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:31.960542 4468 heartbeater.cc:507] Master 127.2.182.62:36531 requested a full tablet report, sending...
I20250904 19:58:31.963698 4280 ts_manager.cc:194] Registered new tserver with Master: 92dbb6eab71a4095a7f207ce564314a3 (127.2.182.1:43243)
I20250904 19:58:31.966454 4280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:58117
W20250904 19:58:32.235502 4472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:32.235973 4472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:32.236505 4472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:32.268013 4472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:32.268831 4472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 19:58:32.302610 4472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:32.303853 4472 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:32.305351 4472 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:32.316612 4478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:32.970503 4468 heartbeater.cc:499] Master 127.2.182.62:36531 was elected leader, sending a full tablet report...
W20250904 19:58:32.316998 4479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:33.407513 4481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:33.410288 4480 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1092 milliseconds
W20250904 19:58:33.410967 4472 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.094s user 0.379s sys 0.716s
W20250904 19:58:33.411208 4472 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.095s user 0.379s sys 0.716s
I20250904 19:58:33.411402 4472 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:33.412355 4472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:33.414494 4472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:33.415850 4472 hybrid_clock.cc:648] HybridClock initialized: now 1757015913415816 us; error 42 us; skew 500 ppm
I20250904 19:58:33.416594 4472 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:33.423682 4472 webserver.cc:480] Webserver started at http://127.2.182.2:36337/ using document root <none> and password file <none>
I20250904 19:58:33.424630 4472 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:33.424837 4472 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:33.425253 4472 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:33.429555 4472 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4867d7394bae4648a2dd8f569627b890"
format_stamp: "Formatted at 2025-09-04 19:58:33 on dist-test-slave-tjsx"
I20250904 19:58:33.430586 4472 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4867d7394bae4648a2dd8f569627b890"
format_stamp: "Formatted at 2025-09-04 19:58:33 on dist-test-slave-tjsx"
I20250904 19:58:33.437958 4472 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250904 19:58:33.443557 4488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:33.444625 4472 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250904 19:58:33.444950 4472 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4867d7394bae4648a2dd8f569627b890"
format_stamp: "Formatted at 2025-09-04 19:58:33 on dist-test-slave-tjsx"
I20250904 19:58:33.445250 4472 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:33.509039 4472 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:33.510416 4472 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:33.510845 4472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:33.513201 4472 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:33.517086 4472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:33.517264 4472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:33.517496 4472 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:33.517642 4472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:33.647150 4472 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:38069
I20250904 19:58:33.647241 4600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:38069 every 8 connection(s)
I20250904 19:58:33.649722 4472 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250904 19:58:33.659509 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 4472
I20250904 19:58:33.660001 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250904 19:58:33.666564 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36531
--builtin_ntp_servers=127.2.182.20:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:58:33.670805 4601 heartbeater.cc:344] Connected to a master server at 127.2.182.62:36531
I20250904 19:58:33.671216 4601 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:33.672204 4601 heartbeater.cc:507] Master 127.2.182.62:36531 requested a full tablet report, sending...
I20250904 19:58:33.674260 4280 ts_manager.cc:194] Registered new tserver with Master: 4867d7394bae4648a2dd8f569627b890 (127.2.182.2:38069)
I20250904 19:58:33.675515 4280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:33147
W20250904 19:58:33.965809 4605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:33.966293 4605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:33.966831 4605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:33.998226 4605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:33.999229 4605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 19:58:34.033481 4605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36531
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:34.034873 4605 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:34.036412 4605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:34.047643 4611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:34.679064 4601 heartbeater.cc:499] Master 127.2.182.62:36531 was elected leader, sending a full tablet report...
W20250904 19:58:34.048038 4612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:35.336989 4614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:35.341137 4605 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.294s user 0.545s sys 0.740s
W20250904 19:58:35.341568 4605 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.294s user 0.547s sys 0.741s
W20250904 19:58:35.341814 4613 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1292 milliseconds
I20250904 19:58:35.341841 4605 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:35.343298 4605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:35.345891 4605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:35.347364 4605 hybrid_clock.cc:648] HybridClock initialized: now 1757015915347324 us; error 36 us; skew 500 ppm
I20250904 19:58:35.348424 4605 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:35.356783 4605 webserver.cc:480] Webserver started at http://127.2.182.3:43767/ using document root <none> and password file <none>
I20250904 19:58:35.358134 4605 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:35.358418 4605 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:35.359035 4605 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:35.365574 4605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b17122bc1e6144b186b3ba633932bc85"
format_stamp: "Formatted at 2025-09-04 19:58:35 on dist-test-slave-tjsx"
I20250904 19:58:35.367065 4605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b17122bc1e6144b186b3ba633932bc85"
format_stamp: "Formatted at 2025-09-04 19:58:35 on dist-test-slave-tjsx"
I20250904 19:58:35.376233 4605 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250904 19:58:35.383416 4621 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:35.384668 4605 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250904 19:58:35.385053 4605 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b17122bc1e6144b186b3ba633932bc85"
format_stamp: "Formatted at 2025-09-04 19:58:35 on dist-test-slave-tjsx"
I20250904 19:58:35.385488 4605 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:35.448444 4605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:35.449858 4605 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:35.450245 4605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:35.452633 4605 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:35.456494 4605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:35.456681 4605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:35.456889 4605 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:35.457018 4605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:35.583212 4605 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:43485
I20250904 19:58:35.583302 4734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:43485 every 8 connection(s)
I20250904 19:58:35.585731 4605 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250904 19:58:35.594059 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 4605
I20250904 19:58:35.594563 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250904 19:58:35.605005 4735 heartbeater.cc:344] Connected to a master server at 127.2.182.62:36531
I20250904 19:58:35.605418 4735 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:35.606359 4735 heartbeater.cc:507] Master 127.2.182.62:36531 requested a full tablet report, sending...
I20250904 19:58:35.608325 4280 ts_manager.cc:194] Registered new tserver with Master: b17122bc1e6144b186b3ba633932bc85 (127.2.182.3:43485)
I20250904 19:58:35.609462 4280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:41091
I20250904 19:58:35.614652 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:58:35.651353 4280 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57990:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250904 19:58:35.669059 4280 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 19:58:35.728825 4403 tablet_service.cc:1468] Processing CreateTablet for tablet 236e0569e51a4fae8843d5b337a17e80 (DEFAULT_TABLE table=TestTable [id=1fdbae3f1c934050be0497b9130124a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:35.728631 4536 tablet_service.cc:1468] Processing CreateTablet for tablet 236e0569e51a4fae8843d5b337a17e80 (DEFAULT_TABLE table=TestTable [id=1fdbae3f1c934050be0497b9130124a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:35.730402 4403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 236e0569e51a4fae8843d5b337a17e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:35.730454 4536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 236e0569e51a4fae8843d5b337a17e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:35.730777 4669 tablet_service.cc:1468] Processing CreateTablet for tablet 236e0569e51a4fae8843d5b337a17e80 (DEFAULT_TABLE table=TestTable [id=1fdbae3f1c934050be0497b9130124a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:35.732753 4669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 236e0569e51a4fae8843d5b337a17e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:35.750131 4754 tablet_bootstrap.cc:492] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3: Bootstrap starting.
I20250904 19:58:35.757164 4755 tablet_bootstrap.cc:492] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890: Bootstrap starting.
I20250904 19:58:35.757745 4754 tablet_bootstrap.cc:654] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:35.758786 4756 tablet_bootstrap.cc:492] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85: Bootstrap starting.
I20250904 19:58:35.760119 4754 log.cc:826] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:35.764163 4755 tablet_bootstrap.cc:654] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:35.764513 4756 tablet_bootstrap.cc:654] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:35.766275 4756 log.cc:826] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:35.766566 4754 tablet_bootstrap.cc:492] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3: No bootstrap required, opened a new log
I20250904 19:58:35.766638 4755 log.cc:826] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:35.768919 4754 ts_tablet_manager.cc:1397] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.012s
I20250904 19:58:35.771332 4755 tablet_bootstrap.cc:492] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890: No bootstrap required, opened a new log
I20250904 19:58:35.771723 4755 ts_tablet_manager.cc:1397] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250904 19:58:35.771729 4756 tablet_bootstrap.cc:492] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85: No bootstrap required, opened a new log
I20250904 19:58:35.772243 4756 ts_tablet_manager.cc:1397] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250904 19:58:35.789605 4755 raft_consensus.cc:357] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.790297 4755 raft_consensus.cc:383] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:35.790558 4755 raft_consensus.cc:738] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4867d7394bae4648a2dd8f569627b890, State: Initialized, Role: FOLLOWER
I20250904 19:58:35.791391 4755 consensus_queue.cc:260] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.795603 4755 ts_tablet_manager.cc:1428] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890: Time spent starting tablet: real 0.024s user 0.019s sys 0.006s
I20250904 19:58:35.795229 4754 raft_consensus.cc:357] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.796207 4754 raft_consensus.cc:383] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:35.796499 4754 raft_consensus.cc:738] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dbb6eab71a4095a7f207ce564314a3, State: Initialized, Role: FOLLOWER
I20250904 19:58:35.797348 4754 consensus_queue.cc:260] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.798002 4756 raft_consensus.cc:357] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.799104 4756 raft_consensus.cc:383] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:35.799454 4756 raft_consensus.cc:738] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17122bc1e6144b186b3ba633932bc85, State: Initialized, Role: FOLLOWER
I20250904 19:58:35.801421 4754 ts_tablet_manager.cc:1428] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3: Time spent starting tablet: real 0.032s user 0.020s sys 0.011s
I20250904 19:58:35.802021 4756 consensus_queue.cc:260] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.805017 4735 heartbeater.cc:499] Master 127.2.182.62:36531 was elected leader, sending a full tablet report...
I20250904 19:58:35.808629 4756 ts_tablet_manager.cc:1428] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85: Time spent starting tablet: real 0.036s user 0.033s sys 0.000s
W20250904 19:58:35.839834 4736 tablet.cc:2378] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:35.887867 4760 raft_consensus.cc:491] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:58:35.888335 4760 raft_consensus.cc:513] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.890846 4760 leader_election.cc:290] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b17122bc1e6144b186b3ba633932bc85 (127.2.182.3:43485), 92dbb6eab71a4095a7f207ce564314a3 (127.2.182.1:43243)
I20250904 19:58:35.902055 4690 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "236e0569e51a4fae8843d5b337a17e80" candidate_uuid: "4867d7394bae4648a2dd8f569627b890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17122bc1e6144b186b3ba633932bc85" is_pre_election: true
I20250904 19:58:35.902124 4423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "236e0569e51a4fae8843d5b337a17e80" candidate_uuid: "4867d7394bae4648a2dd8f569627b890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dbb6eab71a4095a7f207ce564314a3" is_pre_election: true
I20250904 19:58:35.902766 4690 raft_consensus.cc:2466] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4867d7394bae4648a2dd8f569627b890 in term 0.
I20250904 19:58:35.902854 4423 raft_consensus.cc:2466] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4867d7394bae4648a2dd8f569627b890 in term 0.
I20250904 19:58:35.903889 4490 leader_election.cc:304] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4867d7394bae4648a2dd8f569627b890, b17122bc1e6144b186b3ba633932bc85; no voters:
I20250904 19:58:35.904621 4760 raft_consensus.cc:2802] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 19:58:35.904884 4760 raft_consensus.cc:491] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 19:58:35.905135 4760 raft_consensus.cc:3058] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Advancing to term 1
W20250904 19:58:35.906042 4602 tablet.cc:2378] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:35.909842 4760 raft_consensus.cc:513] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.911227 4760 leader_election.cc:290] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [CANDIDATE]: Term 1 election: Requested vote from peers b17122bc1e6144b186b3ba633932bc85 (127.2.182.3:43485), 92dbb6eab71a4095a7f207ce564314a3 (127.2.182.1:43243)
I20250904 19:58:35.912009 4690 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "236e0569e51a4fae8843d5b337a17e80" candidate_uuid: "4867d7394bae4648a2dd8f569627b890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17122bc1e6144b186b3ba633932bc85"
I20250904 19:58:35.912047 4423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "236e0569e51a4fae8843d5b337a17e80" candidate_uuid: "4867d7394bae4648a2dd8f569627b890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dbb6eab71a4095a7f207ce564314a3"
I20250904 19:58:35.912413 4690 raft_consensus.cc:3058] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:35.912441 4423 raft_consensus.cc:3058] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:35.916755 4423 raft_consensus.cc:2466] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4867d7394bae4648a2dd8f569627b890 in term 1.
I20250904 19:58:35.917061 4690 raft_consensus.cc:2466] T 236e0569e51a4fae8843d5b337a17e80 P b17122bc1e6144b186b3ba633932bc85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4867d7394bae4648a2dd8f569627b890 in term 1.
I20250904 19:58:35.917683 4492 leader_election.cc:304] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4867d7394bae4648a2dd8f569627b890, 92dbb6eab71a4095a7f207ce564314a3; no voters:
I20250904 19:58:35.918383 4760 raft_consensus.cc:2802] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:35.919863 4760 raft_consensus.cc:695] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [term 1 LEADER]: Becoming Leader. State: Replica: 4867d7394bae4648a2dd8f569627b890, State: Running, Role: LEADER
I20250904 19:58:35.920675 4760 consensus_queue.cc:237] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } }
I20250904 19:58:35.931008 4280 catalog_manager.cc:5582] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4867d7394bae4648a2dd8f569627b890 (127.2.182.2). New cstate: current_term: 1 leader_uuid: "4867d7394bae4648a2dd8f569627b890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } health_report { overall_health: HEALTHY } } }
W20250904 19:58:35.939008 4469 tablet.cc:2378] T 236e0569e51a4fae8843d5b337a17e80 P 92dbb6eab71a4095a7f207ce564314a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:35.956077 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:58:35.959342 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 92dbb6eab71a4095a7f207ce564314a3 to finish bootstrapping
I20250904 19:58:35.971429 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4867d7394bae4648a2dd8f569627b890 to finish bootstrapping
I20250904 19:58:35.982190 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b17122bc1e6144b186b3ba633932bc85 to finish bootstrapping
I20250904 19:58:35.995596 4280 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57990:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250904 19:58:35.997114 4280 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 19:58:36.013687 4403 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0d4dfb6e584f909afb2cbfa541a627 (DEFAULT_TABLE table=TestAnotherTable [id=f61b8d9d5f594eebad8fe42fb5053455]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250904 19:58:36.014117 4536 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0d4dfb6e584f909afb2cbfa541a627 (DEFAULT_TABLE table=TestAnotherTable [id=f61b8d9d5f594eebad8fe42fb5053455]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250904 19:58:36.014887 4403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0d4dfb6e584f909afb2cbfa541a627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:36.014941 4669 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0d4dfb6e584f909afb2cbfa541a627 (DEFAULT_TABLE table=TestAnotherTable [id=f61b8d9d5f594eebad8fe42fb5053455]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250904 19:58:36.015162 4536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0d4dfb6e584f909afb2cbfa541a627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:36.015765 4669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0d4dfb6e584f909afb2cbfa541a627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:36.026300 4756 tablet_bootstrap.cc:492] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85: Bootstrap starting.
I20250904 19:58:36.028935 4755 tablet_bootstrap.cc:492] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890: Bootstrap starting.
I20250904 19:58:36.029874 4754 tablet_bootstrap.cc:492] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3: Bootstrap starting.
I20250904 19:58:36.032802 4756 tablet_bootstrap.cc:654] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:36.034260 4755 tablet_bootstrap.cc:654] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:36.035543 4754 tablet_bootstrap.cc:654] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:36.038519 4756 tablet_bootstrap.cc:492] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85: No bootstrap required, opened a new log
I20250904 19:58:36.039059 4756 ts_tablet_manager.cc:1397] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20250904 19:58:36.041138 4754 tablet_bootstrap.cc:492] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3: No bootstrap required, opened a new log
I20250904 19:58:36.041249 4756 raft_consensus.cc:357] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.041564 4754 ts_tablet_manager.cc:1397] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250904 19:58:36.041687 4755 tablet_bootstrap.cc:492] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890: No bootstrap required, opened a new log
I20250904 19:58:36.041754 4756 raft_consensus.cc:383] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:36.042013 4756 raft_consensus.cc:738] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17122bc1e6144b186b3ba633932bc85, State: Initialized, Role: FOLLOWER
I20250904 19:58:36.042107 4755 ts_tablet_manager.cc:1397] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20250904 19:58:36.042730 4756 consensus_queue.cc:260] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.044346 4754 raft_consensus.cc:357] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.045106 4754 raft_consensus.cc:383] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:36.044880 4755 raft_consensus.cc:357] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.045454 4754 raft_consensus.cc:738] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92dbb6eab71a4095a7f207ce564314a3, State: Initialized, Role: FOLLOWER
I20250904 19:58:36.045611 4755 raft_consensus.cc:383] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:36.045917 4755 raft_consensus.cc:738] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4867d7394bae4648a2dd8f569627b890, State: Initialized, Role: FOLLOWER
I20250904 19:58:36.046239 4756 ts_tablet_manager.cc:1428] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250904 19:58:36.046357 4754 consensus_queue.cc:260] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.046597 4755 consensus_queue.cc:260] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.049367 4755 ts_tablet_manager.cc:1428] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20250904 19:58:36.053800 4754 ts_tablet_manager.cc:1428] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3: Time spent starting tablet: real 0.012s user 0.000s sys 0.006s
I20250904 19:58:36.185056 4762 raft_consensus.cc:491] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:58:36.185436 4762 raft_consensus.cc:513] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.187549 4762 leader_election.cc:290] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92dbb6eab71a4095a7f207ce564314a3 (127.2.182.1:43243), 4867d7394bae4648a2dd8f569627b890 (127.2.182.2:38069)
I20250904 19:58:36.198185 4423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0d4dfb6e584f909afb2cbfa541a627" candidate_uuid: "b17122bc1e6144b186b3ba633932bc85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dbb6eab71a4095a7f207ce564314a3" is_pre_election: true
I20250904 19:58:36.198683 4423 raft_consensus.cc:2466] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b17122bc1e6144b186b3ba633932bc85 in term 0.
I20250904 19:58:36.199715 4625 leader_election.cc:304] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92dbb6eab71a4095a7f207ce564314a3, b17122bc1e6144b186b3ba633932bc85; no voters:
I20250904 19:58:36.200441 4762 raft_consensus.cc:2802] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 19:58:36.200387 4556 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0d4dfb6e584f909afb2cbfa541a627" candidate_uuid: "b17122bc1e6144b186b3ba633932bc85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4867d7394bae4648a2dd8f569627b890" is_pre_election: true
I20250904 19:58:36.200673 4762 raft_consensus.cc:491] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 19:58:36.200937 4762 raft_consensus.cc:3058] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:36.201011 4556 raft_consensus.cc:2466] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b17122bc1e6144b186b3ba633932bc85 in term 0.
I20250904 19:58:36.205080 4762 raft_consensus.cc:513] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.206358 4762 leader_election.cc:290] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [CANDIDATE]: Term 1 election: Requested vote from peers 92dbb6eab71a4095a7f207ce564314a3 (127.2.182.1:43243), 4867d7394bae4648a2dd8f569627b890 (127.2.182.2:38069)
I20250904 19:58:36.207078 4423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0d4dfb6e584f909afb2cbfa541a627" candidate_uuid: "b17122bc1e6144b186b3ba633932bc85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92dbb6eab71a4095a7f207ce564314a3"
I20250904 19:58:36.207202 4556 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0d4dfb6e584f909afb2cbfa541a627" candidate_uuid: "b17122bc1e6144b186b3ba633932bc85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4867d7394bae4648a2dd8f569627b890"
I20250904 19:58:36.207465 4423 raft_consensus.cc:3058] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:36.207625 4556 raft_consensus.cc:3058] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:36.211434 4423 raft_consensus.cc:2466] T 1b0d4dfb6e584f909afb2cbfa541a627 P 92dbb6eab71a4095a7f207ce564314a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b17122bc1e6144b186b3ba633932bc85 in term 1.
I20250904 19:58:36.211694 4556 raft_consensus.cc:2466] T 1b0d4dfb6e584f909afb2cbfa541a627 P 4867d7394bae4648a2dd8f569627b890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b17122bc1e6144b186b3ba633932bc85 in term 1.
I20250904 19:58:36.212211 4625 leader_election.cc:304] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92dbb6eab71a4095a7f207ce564314a3, b17122bc1e6144b186b3ba633932bc85; no voters:
I20250904 19:58:36.212736 4762 raft_consensus.cc:2802] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:36.214139 4762 raft_consensus.cc:695] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [term 1 LEADER]: Becoming Leader. State: Replica: b17122bc1e6144b186b3ba633932bc85, State: Running, Role: LEADER
I20250904 19:58:36.214849 4762 consensus_queue.cc:237] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } }
I20250904 19:58:36.224447 4278 catalog_manager.cc:5582] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 reported cstate change: term changed from 0 to 1, leader changed from <none> to b17122bc1e6144b186b3ba633932bc85 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "b17122bc1e6144b186b3ba633932bc85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 } health_report { overall_health: HEALTHY } } }
I20250904 19:58:36.330611 4760 consensus_queue.cc:1035] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 19:58:36.344848 4760 consensus_queue.cc:1035] T 236e0569e51a4fae8843d5b337a17e80 P 4867d7394bae4648a2dd8f569627b890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17122bc1e6144b186b3ba633932bc85" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 43485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 19:58:36.559163 4775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:36.559734 4775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:36.590085 4775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250904 19:58:36.614060 4762 consensus_queue.cc:1035] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92dbb6eab71a4095a7f207ce564314a3" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 43243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 19:58:36.639348 4773 consensus_queue.cc:1035] T 1b0d4dfb6e584f909afb2cbfa541a627 P b17122bc1e6144b186b3ba633932bc85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4867d7394bae4648a2dd8f569627b890" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 19:58:37.829794 4775 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.154s user 0.476s sys 0.674s
W20250904 19:58:37.830166 4775 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.155s user 0.476s sys 0.674s
W20250904 19:58:39.186861 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:39.187386 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:39.217593 4802 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250904 19:58:40.400724 4802 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.147s user 0.502s sys 0.642s
W20250904 19:58:40.401093 4802 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.147s user 0.502s sys 0.642s
W20250904 19:58:41.782578 4820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:41.783130 4820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:41.816301 4820 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250904 19:58:42.979045 4820 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.128s user 0.513s sys 0.614s
W20250904 19:58:42.979322 4820 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.128s user 0.513s sys 0.614s
W20250904 19:58:44.322335 4835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:44.322921 4835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:44.352712 4835 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250904 19:58:45.531262 4835 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.145s user 0.458s sys 0.686s
W20250904 19:58:45.531551 4835 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.145s user 0.458s sys 0.686s
I20250904 19:58:46.593019 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 4339
I20250904 19:58:46.617115 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 4472
I20250904 19:58:46.637460 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 4605
I20250904 19:58:46.657035 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 4247
2025-09-04T19:58:46Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18811 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250904 19:58:46.707049 2776 test_util.cc:276] Using random seed: 640488273
I20250904 19:58:46.710929 2776 ts_itest-base.cc:115] Starting cluster with:
I20250904 19:58:46.711097 2776 ts_itest-base.cc:116] --------------
I20250904 19:58:46.711231 2776 ts_itest-base.cc:117] 3 tablet servers
I20250904 19:58:46.711359 2776 ts_itest-base.cc:118] 3 replicas per TS
I20250904 19:58:46.711491 2776 ts_itest-base.cc:119] --------------
2025-09-04T19:58:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T19:58:46Z Disabled control of system clock
I20250904 19:58:46.742033 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36565
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:40915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:36565
--superuser_acl=no-such-user with env {}
W20250904 19:58:47.024942 4856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:47.025468 4856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:47.025902 4856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:47.055784 4856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:58:47.056043 4856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:47.056273 4856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:58:47.056485 4856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:58:47.089912 4856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:36565
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:36565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:47.091065 4856 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:47.092547 4856 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:47.103207 4862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:47.103281 4863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:48.339341 4865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:48.342284 4864 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1237 milliseconds
W20250904 19:58:48.342862 4856 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.240s user 0.472s sys 0.758s
W20250904 19:58:48.343098 4856 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.240s user 0.472s sys 0.758s
I20250904 19:58:48.343271 4856 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:48.344286 4856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:48.351207 4856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:48.352543 4856 hybrid_clock.cc:648] HybridClock initialized: now 1757015928352496 us; error 55 us; skew 500 ppm
I20250904 19:58:48.353385 4856 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:48.360014 4856 webserver.cc:480] Webserver started at http://127.2.182.62:43415/ using document root <none> and password file <none>
I20250904 19:58:48.360880 4856 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:48.361085 4856 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:48.361492 4856 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:48.365662 4856 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "323151bd1e8b42ec80abcc35b16f6814"
format_stamp: "Formatted at 2025-09-04 19:58:48 on dist-test-slave-tjsx"
I20250904 19:58:48.366780 4856 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "323151bd1e8b42ec80abcc35b16f6814"
format_stamp: "Formatted at 2025-09-04 19:58:48 on dist-test-slave-tjsx"
I20250904 19:58:48.373921 4856 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250904 19:58:48.379410 4872 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:48.380424 4856 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250904 19:58:48.380733 4856 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "323151bd1e8b42ec80abcc35b16f6814"
format_stamp: "Formatted at 2025-09-04 19:58:48 on dist-test-slave-tjsx"
I20250904 19:58:48.381021 4856 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:48.437554 4856 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:48.438959 4856 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:48.439363 4856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:48.501958 4856 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:36565
I20250904 19:58:48.502030 4923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:36565 every 8 connection(s)
I20250904 19:58:48.504421 4856 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:58:48.509485 4924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:48.512301 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 4856
I20250904 19:58:48.512761 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250904 19:58:48.529794 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814: Bootstrap starting.
I20250904 19:58:48.535202 4924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:48.537132 4924 log.cc:826] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:48.540872 4924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814: No bootstrap required, opened a new log
I20250904 19:58:48.557163 4924 raft_consensus.cc:357] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323151bd1e8b42ec80abcc35b16f6814" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36565 } }
I20250904 19:58:48.557834 4924 raft_consensus.cc:383] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:48.558109 4924 raft_consensus.cc:738] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323151bd1e8b42ec80abcc35b16f6814, State: Initialized, Role: FOLLOWER
I20250904 19:58:48.558763 4924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323151bd1e8b42ec80abcc35b16f6814" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36565 } }
I20250904 19:58:48.559255 4924 raft_consensus.cc:397] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:58:48.559511 4924 raft_consensus.cc:491] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:58:48.559787 4924 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:48.563680 4924 raft_consensus.cc:513] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323151bd1e8b42ec80abcc35b16f6814" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36565 } }
I20250904 19:58:48.564317 4924 leader_election.cc:304] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 323151bd1e8b42ec80abcc35b16f6814; no voters:
I20250904 19:58:48.565845 4924 leader_election.cc:290] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:58:48.566531 4929 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:48.568429 4929 raft_consensus.cc:695] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [term 1 LEADER]: Becoming Leader. State: Replica: 323151bd1e8b42ec80abcc35b16f6814, State: Running, Role: LEADER
I20250904 19:58:48.569090 4929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323151bd1e8b42ec80abcc35b16f6814" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36565 } }
I20250904 19:58:48.570004 4924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:58:48.575220 4930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "323151bd1e8b42ec80abcc35b16f6814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323151bd1e8b42ec80abcc35b16f6814" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36565 } } }
I20250904 19:58:48.576078 4930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:48.576608 4931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 323151bd1e8b42ec80abcc35b16f6814. Latest consensus state: current_term: 1 leader_uuid: "323151bd1e8b42ec80abcc35b16f6814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "323151bd1e8b42ec80abcc35b16f6814" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 36565 } } }
I20250904 19:58:48.577139 4931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814 [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:48.583935 4936 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:58:48.594439 4936 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:58:48.608420 4936 catalog_manager.cc:1349] Generated new cluster ID: 22a63e4a39d145f7b0682415e6c80445
I20250904 19:58:48.608686 4936 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:58:48.621730 4936 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 19:58:48.623343 4936 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:58:48.633533 4936 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 323151bd1e8b42ec80abcc35b16f6814: Generated new TSK 0
I20250904 19:58:48.634305 4936 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 19:58:48.655791 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36565
--builtin_ntp_servers=127.2.182.20:40915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250904 19:58:48.940316 4948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:48.940989 4948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:48.941695 4948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:48.972440 4948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:48.973241 4948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 19:58:49.005988 4948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:49.007469 4948 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:49.008999 4948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:49.020531 4954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:49.021322 4955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:50.289014 4948 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.270s user 0.533s sys 0.735s
W20250904 19:58:50.289387 4948 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.270s user 0.534s sys 0.735s
I20250904 19:58:50.295157 4948 server_base.cc:1047] running on GCE node
W20250904 19:58:50.296825 4959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:50.298188 4948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:50.307183 4948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:50.308704 4948 hybrid_clock.cc:648] HybridClock initialized: now 1757015930308636 us; error 77 us; skew 500 ppm
I20250904 19:58:50.309724 4948 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:50.317854 4948 webserver.cc:480] Webserver started at http://127.2.182.1:43573/ using document root <none> and password file <none>
I20250904 19:58:50.319180 4948 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:50.319449 4948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:50.319979 4948 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:50.326370 4948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "078a94c9efb64a1988a91c42811388e4"
format_stamp: "Formatted at 2025-09-04 19:58:50 on dist-test-slave-tjsx"
I20250904 19:58:50.327816 4948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "078a94c9efb64a1988a91c42811388e4"
format_stamp: "Formatted at 2025-09-04 19:58:50 on dist-test-slave-tjsx"
I20250904 19:58:50.336870 4948 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250904 19:58:50.344009 4964 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:50.345197 4948 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250904 19:58:50.345590 4948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "078a94c9efb64a1988a91c42811388e4"
format_stamp: "Formatted at 2025-09-04 19:58:50 on dist-test-slave-tjsx"
I20250904 19:58:50.345978 4948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:50.430183 4948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:50.431684 4948 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:50.432108 4948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:50.434645 4948 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:50.439090 4948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:50.439284 4948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:50.439548 4948 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:50.439699 4948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:50.589306 4948 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36049
I20250904 19:58:50.589416 5076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:36049 every 8 connection(s)
I20250904 19:58:50.591784 4948 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 19:58:50.597594 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 4948
I20250904 19:58:50.597956 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250904 19:58:50.604645 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36565
--builtin_ntp_servers=127.2.182.20:40915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:58:50.618055 5077 heartbeater.cc:344] Connected to a master server at 127.2.182.62:36565
I20250904 19:58:50.618427 5077 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:50.619375 5077 heartbeater.cc:507] Master 127.2.182.62:36565 requested a full tablet report, sending...
I20250904 19:58:50.621573 4889 ts_manager.cc:194] Registered new tserver with Master: 078a94c9efb64a1988a91c42811388e4 (127.2.182.1:36049)
I20250904 19:58:50.623510 4889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:33457
W20250904 19:58:50.913072 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:50.913544 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:50.914041 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:50.944689 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:50.945529 5081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 19:58:50.978510 5081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:50.979730 5081 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:50.981209 5081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:50.992022 5087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:51.626412 5077 heartbeater.cc:499] Master 127.2.182.62:36565 was elected leader, sending a full tablet report...
W20250904 19:58:50.996045 5090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:50.992835 5088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:52.053917 5081 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250904 19:58:52.054296 5089 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250904 19:58:52.057961 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:52.060492 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:52.061897 5081 hybrid_clock.cc:648] HybridClock initialized: now 1757015932061863 us; error 54 us; skew 500 ppm
I20250904 19:58:52.062744 5081 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:52.068967 5081 webserver.cc:480] Webserver started at http://127.2.182.2:39865/ using document root <none> and password file <none>
I20250904 19:58:52.069836 5081 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:52.070042 5081 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:52.070490 5081 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:52.076254 5081 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f349db058f704afc936d32dddac10466"
format_stamp: "Formatted at 2025-09-04 19:58:52 on dist-test-slave-tjsx"
I20250904 19:58:52.077304 5081 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f349db058f704afc936d32dddac10466"
format_stamp: "Formatted at 2025-09-04 19:58:52 on dist-test-slave-tjsx"
I20250904 19:58:52.084229 5081 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250904 19:58:52.089560 5097 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:52.090482 5081 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250904 19:58:52.090804 5081 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f349db058f704afc936d32dddac10466"
format_stamp: "Formatted at 2025-09-04 19:58:52 on dist-test-slave-tjsx"
I20250904 19:58:52.091123 5081 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:52.138983 5081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:52.140321 5081 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:52.140740 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:52.143074 5081 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:52.146890 5081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:52.147085 5081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:52.147317 5081 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:52.147476 5081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:52.269078 5081 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:45197
I20250904 19:58:52.269197 5209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:45197 every 8 connection(s)
I20250904 19:58:52.271512 5081 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250904 19:58:52.277420 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 5081
I20250904 19:58:52.277784 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250904 19:58:52.283946 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36565
--builtin_ntp_servers=127.2.182.20:40915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:58:52.291327 5210 heartbeater.cc:344] Connected to a master server at 127.2.182.62:36565
I20250904 19:58:52.291777 5210 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:52.292929 5210 heartbeater.cc:507] Master 127.2.182.62:36565 requested a full tablet report, sending...
I20250904 19:58:52.295162 4889 ts_manager.cc:194] Registered new tserver with Master: f349db058f704afc936d32dddac10466 (127.2.182.2:45197)
I20250904 19:58:52.296401 4889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:44801
W20250904 19:58:52.567239 5214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:52.567708 5214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:52.568171 5214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:52.598582 5214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:52.599426 5214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 19:58:52.633317 5214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:40915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:36565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:52.634538 5214 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:52.636031 5214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:52.647166 5220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:58:53.299733 5210 heartbeater.cc:499] Master 127.2.182.62:36565 was elected leader, sending a full tablet report...
W20250904 19:58:54.049728 5219 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 5214
W20250904 19:58:54.113893 5219 kernel_stack_watchdog.cc:198] Thread 5214 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 19:58:52.647908 5221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:54.114606 5214 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.467s user 0.670s sys 0.794s
W20250904 19:58:54.114985 5214 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.468s user 0.670s sys 0.794s
W20250904 19:58:54.116299 5223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:54.118861 5222 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1467 milliseconds
I20250904 19:58:54.118925 5214 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:54.120024 5214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:54.121874 5214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:54.123174 5214 hybrid_clock.cc:648] HybridClock initialized: now 1757015934123138 us; error 38 us; skew 500 ppm
I20250904 19:58:54.123950 5214 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:54.129370 5214 webserver.cc:480] Webserver started at http://127.2.182.3:40715/ using document root <none> and password file <none>
I20250904 19:58:54.130173 5214 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:54.130344 5214 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:54.130744 5214 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:54.134842 5214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a3f142d17e534a21b19aefb8e879688c"
format_stamp: "Formatted at 2025-09-04 19:58:54 on dist-test-slave-tjsx"
I20250904 19:58:54.136034 5214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a3f142d17e534a21b19aefb8e879688c"
format_stamp: "Formatted at 2025-09-04 19:58:54 on dist-test-slave-tjsx"
I20250904 19:58:54.142396 5214 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250904 19:58:54.147482 5230 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:54.148388 5214 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250904 19:58:54.148684 5214 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a3f142d17e534a21b19aefb8e879688c"
format_stamp: "Formatted at 2025-09-04 19:58:54 on dist-test-slave-tjsx"
I20250904 19:58:54.148973 5214 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:54.205950 5214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:54.207362 5214 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:54.207765 5214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:54.210125 5214 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:58:54.214102 5214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:58:54.214352 5214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:54.214588 5214 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:58:54.214772 5214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250904 19:58:54.340276 5214 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:46251
I20250904 19:58:54.340379 5342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:46251 every 8 connection(s)
I20250904 19:58:54.342604 5214 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250904 19:58:54.344439 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 5214
I20250904 19:58:54.344900 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250904 19:58:54.361383 5343 heartbeater.cc:344] Connected to a master server at 127.2.182.62:36565
I20250904 19:58:54.361766 5343 heartbeater.cc:461] Registering TS with master...
I20250904 19:58:54.362926 5343 heartbeater.cc:507] Master 127.2.182.62:36565 requested a full tablet report, sending...
I20250904 19:58:54.364910 4889 ts_manager.cc:194] Registered new tserver with Master: a3f142d17e534a21b19aefb8e879688c (127.2.182.3:46251)
I20250904 19:58:54.366052 4889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:60781
I20250904 19:58:54.377992 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:58:54.405406 4889 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33136:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250904 19:58:54.423488 4889 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 19:58:54.476529 5278 tablet_service.cc:1468] Processing CreateTablet for tablet f01d063f0a454a1c8c622e861b561fe2 (DEFAULT_TABLE table=TestTable [id=ffab0c07f8e7426a9b217d6979ef9227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:54.478287 5278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01d063f0a454a1c8c622e861b561fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:54.478894 5012 tablet_service.cc:1468] Processing CreateTablet for tablet f01d063f0a454a1c8c622e861b561fe2 (DEFAULT_TABLE table=TestTable [id=ffab0c07f8e7426a9b217d6979ef9227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:54.480207 5012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01d063f0a454a1c8c622e861b561fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:54.479751 5145 tablet_service.cc:1468] Processing CreateTablet for tablet f01d063f0a454a1c8c622e861b561fe2 (DEFAULT_TABLE table=TestTable [id=ffab0c07f8e7426a9b217d6979ef9227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:58:54.481353 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f01d063f0a454a1c8c622e861b561fe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:54.501430 5362 tablet_bootstrap.cc:492] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c: Bootstrap starting.
I20250904 19:58:54.510730 5362 tablet_bootstrap.cc:654] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:54.513046 5364 tablet_bootstrap.cc:492] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466: Bootstrap starting.
I20250904 19:58:54.513513 5362 log.cc:826] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:54.517689 5363 tablet_bootstrap.cc:492] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4: Bootstrap starting.
I20250904 19:58:54.519990 5364 tablet_bootstrap.cc:654] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:54.522178 5364 log.cc:826] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:54.523500 5362 tablet_bootstrap.cc:492] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c: No bootstrap required, opened a new log
I20250904 19:58:54.523988 5362 ts_tablet_manager.cc:1397] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.006s
I20250904 19:58:54.526935 5363 tablet_bootstrap.cc:654] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:54.529218 5363 log.cc:826] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:54.534571 5364 tablet_bootstrap.cc:492] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466: No bootstrap required, opened a new log
I20250904 19:58:54.535086 5364 ts_tablet_manager.cc:1397] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.009s
I20250904 19:58:54.540779 5363 tablet_bootstrap.cc:492] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4: No bootstrap required, opened a new log
I20250904 19:58:54.541277 5363 ts_tablet_manager.cc:1397] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.012s
I20250904 19:58:54.550555 5362 raft_consensus.cc:357] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.551456 5362 raft_consensus.cc:383] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:54.551761 5362 raft_consensus.cc:738] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f142d17e534a21b19aefb8e879688c, State: Initialized, Role: FOLLOWER
I20250904 19:58:54.552747 5362 consensus_queue.cc:260] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.562474 5364 raft_consensus.cc:357] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.563493 5364 raft_consensus.cc:383] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:54.563843 5364 raft_consensus.cc:738] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f349db058f704afc936d32dddac10466, State: Initialized, Role: FOLLOWER
I20250904 19:58:54.564962 5364 consensus_queue.cc:260] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.566990 5343 heartbeater.cc:499] Master 127.2.182.62:36565 was elected leader, sending a full tablet report...
I20250904 19:58:54.567998 5362 ts_tablet_manager.cc:1428] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c: Time spent starting tablet: real 0.044s user 0.030s sys 0.012s
I20250904 19:58:54.571235 5364 ts_tablet_manager.cc:1428] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466: Time spent starting tablet: real 0.036s user 0.031s sys 0.001s
I20250904 19:58:54.571130 5363 raft_consensus.cc:357] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.571985 5363 raft_consensus.cc:383] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:54.572271 5363 raft_consensus.cc:738] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078a94c9efb64a1988a91c42811388e4, State: Initialized, Role: FOLLOWER
I20250904 19:58:54.573132 5363 consensus_queue.cc:260] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.578536 5363 ts_tablet_manager.cc:1428] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4: Time spent starting tablet: real 0.037s user 0.030s sys 0.000s
W20250904 19:58:54.597595 5344 tablet.cc:2378] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 19:58:54.600677 5078 tablet.cc:2378] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:54.670146 5368 raft_consensus.cc:491] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:58:54.670596 5368 raft_consensus.cc:513] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.672832 5368 leader_election.cc:290] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078a94c9efb64a1988a91c42811388e4 (127.2.182.1:36049), f349db058f704afc936d32dddac10466 (127.2.182.2:45197)
I20250904 19:58:54.682179 5032 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f01d063f0a454a1c8c622e861b561fe2" candidate_uuid: "a3f142d17e534a21b19aefb8e879688c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078a94c9efb64a1988a91c42811388e4" is_pre_election: true
I20250904 19:58:54.683023 5032 raft_consensus.cc:2466] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3f142d17e534a21b19aefb8e879688c in term 0.
I20250904 19:58:54.683331 5369 raft_consensus.cc:491] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:58:54.683831 5165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f01d063f0a454a1c8c622e861b561fe2" candidate_uuid: "a3f142d17e534a21b19aefb8e879688c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f349db058f704afc936d32dddac10466" is_pre_election: true
I20250904 19:58:54.683825 5369 raft_consensus.cc:513] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.684358 5231 leader_election.cc:304] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078a94c9efb64a1988a91c42811388e4, a3f142d17e534a21b19aefb8e879688c; no voters:
I20250904 19:58:54.685132 5368 raft_consensus.cc:2802] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 19:58:54.685119 5165 raft_consensus.cc:2466] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3f142d17e534a21b19aefb8e879688c in term 0.
I20250904 19:58:54.685485 5368 raft_consensus.cc:491] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 19:58:54.685786 5368 raft_consensus.cc:3058] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:54.689294 5369 leader_election.cc:290] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3f142d17e534a21b19aefb8e879688c (127.2.182.3:46251), 078a94c9efb64a1988a91c42811388e4 (127.2.182.1:36049)
I20250904 19:58:54.692273 5368 raft_consensus.cc:513] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.694416 5368 leader_election.cc:290] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [CANDIDATE]: Term 1 election: Requested vote from peers 078a94c9efb64a1988a91c42811388e4 (127.2.182.1:36049), f349db058f704afc936d32dddac10466 (127.2.182.2:45197)
I20250904 19:58:54.694487 5032 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f01d063f0a454a1c8c622e861b561fe2" candidate_uuid: "a3f142d17e534a21b19aefb8e879688c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078a94c9efb64a1988a91c42811388e4"
I20250904 19:58:54.694629 5165 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f01d063f0a454a1c8c622e861b561fe2" candidate_uuid: "a3f142d17e534a21b19aefb8e879688c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f349db058f704afc936d32dddac10466"
I20250904 19:58:54.695024 5032 raft_consensus.cc:3058] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:54.695114 5165 raft_consensus.cc:3058] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:54.699175 5298 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f01d063f0a454a1c8c622e861b561fe2" candidate_uuid: "f349db058f704afc936d32dddac10466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3f142d17e534a21b19aefb8e879688c" is_pre_election: true
I20250904 19:58:54.700071 5298 raft_consensus.cc:2391] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f349db058f704afc936d32dddac10466 in current term 1: Already voted for candidate a3f142d17e534a21b19aefb8e879688c in this term.
I20250904 19:58:54.700634 5165 raft_consensus.cc:2466] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3f142d17e534a21b19aefb8e879688c in term 1.
I20250904 19:58:54.701013 5032 raft_consensus.cc:2466] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3f142d17e534a21b19aefb8e879688c in term 1.
I20250904 19:58:54.701694 5234 leader_election.cc:304] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3f142d17e534a21b19aefb8e879688c, f349db058f704afc936d32dddac10466; no voters:
I20250904 19:58:54.702538 5368 raft_consensus.cc:2802] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:54.703773 5032 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f01d063f0a454a1c8c622e861b561fe2" candidate_uuid: "f349db058f704afc936d32dddac10466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078a94c9efb64a1988a91c42811388e4" is_pre_election: true
I20250904 19:58:54.704313 5032 raft_consensus.cc:2391] T f01d063f0a454a1c8c622e861b561fe2 P 078a94c9efb64a1988a91c42811388e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f349db058f704afc936d32dddac10466 in current term 1: Already voted for candidate a3f142d17e534a21b19aefb8e879688c in this term.
I20250904 19:58:54.704363 5368 raft_consensus.cc:695] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [term 1 LEADER]: Becoming Leader. State: Replica: a3f142d17e534a21b19aefb8e879688c, State: Running, Role: LEADER
I20250904 19:58:54.705268 5098 leader_election.cc:304] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f349db058f704afc936d32dddac10466; no voters: 078a94c9efb64a1988a91c42811388e4, a3f142d17e534a21b19aefb8e879688c
I20250904 19:58:54.705227 5368 consensus_queue.cc:237] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } }
I20250904 19:58:54.705915 5369 raft_consensus.cc:2747] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250904 19:58:54.715706 4889 catalog_manager.cc:5582] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c reported cstate change: term changed from 0 to 1, leader changed from <none> to a3f142d17e534a21b19aefb8e879688c (127.2.182.3). New cstate: current_term: 1 leader_uuid: "a3f142d17e534a21b19aefb8e879688c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3f142d17e534a21b19aefb8e879688c" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 46251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 } health_report { overall_health: UNKNOWN } } }
W20250904 19:58:54.777915 5211 tablet.cc:2378] T f01d063f0a454a1c8c622e861b561fe2 P f349db058f704afc936d32dddac10466: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 19:58:54.780756 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:58:54.783829 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 078a94c9efb64a1988a91c42811388e4 to finish bootstrapping
I20250904 19:58:54.795982 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f349db058f704afc936d32dddac10466 to finish bootstrapping
I20250904 19:58:54.806325 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a3f142d17e534a21b19aefb8e879688c to finish bootstrapping
I20250904 19:58:55.176396 5368 consensus_queue.cc:1035] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [LEADER]: Connected to new peer: Peer: permanent_uuid: "078a94c9efb64a1988a91c42811388e4" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 19:58:55.198840 5374 consensus_queue.cc:1035] T f01d063f0a454a1c8c622e861b561fe2 P a3f142d17e534a21b19aefb8e879688c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f349db058f704afc936d32dddac10466" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 19:58:56.321760 4889 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:33154
I20250904 19:58:57.398048 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 4948
I20250904 19:58:57.417759 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 5081
I20250904 19:58:57.437100 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 5214
I20250904 19:58:57.456112 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 4856
2025-09-04T19:58:57Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10799 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250904 19:58:57.506739 2776 test_util.cc:276] Using random seed: 651287962
I20250904 19:58:57.510532 2776 ts_itest-base.cc:115] Starting cluster with:
I20250904 19:58:57.510684 2776 ts_itest-base.cc:116] --------------
I20250904 19:58:57.510835 2776 ts_itest-base.cc:117] 3 tablet servers
I20250904 19:58:57.510941 2776 ts_itest-base.cc:118] 3 replicas per TS
I20250904 19:58:57.511041 2776 ts_itest-base.cc:119] --------------
2025-09-04T19:58:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T19:58:57Z Disabled control of system clock
I20250904 19:58:57.541415 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38495
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38495 with env {}
W20250904 19:58:57.823431 5411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:57.823956 5411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:57.824357 5411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:57.854472 5411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:58:57.854791 5411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:57.855003 5411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:58:57.855193 5411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:58:57.888885 5411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38495
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:57.890074 5411 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:57.891734 5411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:57.901187 5417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:57.902419 5418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:59.010236 5420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:58:59.012432 5419 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1107 milliseconds
I20250904 19:58:59.012574 5411 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:58:59.013790 5411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:58:59.016142 5411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:58:59.017462 5411 hybrid_clock.cc:648] HybridClock initialized: now 1757015939017425 us; error 44 us; skew 500 ppm
I20250904 19:58:59.018216 5411 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:58:59.024199 5411 webserver.cc:480] Webserver started at http://127.2.182.62:43493/ using document root <none> and password file <none>
I20250904 19:58:59.025058 5411 fs_manager.cc:362] Metadata directory not provided
I20250904 19:58:59.025275 5411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:58:59.025712 5411 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:58:59.029853 5411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "116b311348744b64b8e53eec1aad5435"
format_stamp: "Formatted at 2025-09-04 19:58:59 on dist-test-slave-tjsx"
I20250904 19:58:59.030928 5411 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "116b311348744b64b8e53eec1aad5435"
format_stamp: "Formatted at 2025-09-04 19:58:59 on dist-test-slave-tjsx"
I20250904 19:58:59.037689 5411 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250904 19:58:59.042789 5427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:58:59.043706 5411 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250904 19:58:59.043998 5411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "116b311348744b64b8e53eec1aad5435"
format_stamp: "Formatted at 2025-09-04 19:58:59 on dist-test-slave-tjsx"
I20250904 19:58:59.044291 5411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:58:59.088866 5411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:58:59.090222 5411 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:58:59.090629 5411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:58:59.155236 5411 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38495
I20250904 19:58:59.155298 5478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:38495 every 8 connection(s)
I20250904 19:58:59.157732 5411 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:58:59.162351 5479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:58:59.165560 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 5411
I20250904 19:58:59.165920 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250904 19:58:59.183965 5479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap starting.
I20250904 19:58:59.188817 5479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Neither blocks nor log segments found. Creating new log.
I20250904 19:58:59.190663 5479 log.cc:826] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Log is configured to *not* fsync() on all Append() calls
I20250904 19:58:59.195406 5479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: No bootstrap required, opened a new log
I20250904 19:58:59.213119 5479 raft_consensus.cc:357] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:58:59.213819 5479 raft_consensus.cc:383] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:58:59.214057 5479 raft_consensus.cc:738] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116b311348744b64b8e53eec1aad5435, State: Initialized, Role: FOLLOWER
I20250904 19:58:59.214790 5479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:58:59.215282 5479 raft_consensus.cc:397] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:58:59.215543 5479 raft_consensus.cc:491] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:58:59.215817 5479 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:58:59.219648 5479 raft_consensus.cc:513] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:58:59.220283 5479 leader_election.cc:304] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116b311348744b64b8e53eec1aad5435; no voters:
I20250904 19:58:59.221822 5479 leader_election.cc:290] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:58:59.222501 5484 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:58:59.224476 5484 raft_consensus.cc:695] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 1 LEADER]: Becoming Leader. State: Replica: 116b311348744b64b8e53eec1aad5435, State: Running, Role: LEADER
I20250904 19:58:59.225098 5484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:58:59.225582 5479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:58:59.236157 5485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "116b311348744b64b8e53eec1aad5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } } }
I20250904 19:58:59.236501 5486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116b311348744b64b8e53eec1aad5435. Latest consensus state: current_term: 1 leader_uuid: "116b311348744b64b8e53eec1aad5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } } }
I20250904 19:58:59.236828 5485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:59.237160 5486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: This master's current role is: LEADER
I20250904 19:58:59.239737 5494 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:58:59.250329 5494 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:58:59.265946 5494 catalog_manager.cc:1349] Generated new cluster ID: 0fe8823e3bc14e35badbe62ce3e59c2a
I20250904 19:58:59.266232 5494 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:58:59.296020 5494 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 19:58:59.297415 5494 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:58:59.310067 5494 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Generated new TSK 0
I20250904 19:58:59.310928 5494 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 19:58:59.328922 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38495
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250904 19:58:59.617489 5503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:58:59.617959 5503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:58:59.618438 5503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:58:59.649039 5503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:58:59.649930 5503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 19:58:59.684926 5503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:58:59.686182 5503 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:58:59.687773 5503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:58:59.700054 5509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:01.102366 5508 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 5503
W20250904 19:59:01.268777 5508 kernel_stack_watchdog.cc:198] Thread 5503 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 19:58:59.701187 5510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:01.270226 5511 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1568 milliseconds
W20250904 19:59:01.270252 5503 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.569s user 0.604s sys 0.895s
W20250904 19:59:01.270614 5503 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.570s user 0.604s sys 0.895s
W20250904 19:59:01.270743 5512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:01.270925 5503 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:01.275282 5503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:01.277549 5503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:01.278930 5503 hybrid_clock.cc:648] HybridClock initialized: now 1757015941278891 us; error 47 us; skew 500 ppm
I20250904 19:59:01.279683 5503 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:01.285861 5503 webserver.cc:480] Webserver started at http://127.2.182.1:35221/ using document root <none> and password file <none>
I20250904 19:59:01.286743 5503 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:01.286957 5503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:01.287369 5503 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:59:01.291509 5503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "47edf36cc269417ba960d1c1955afde1"
format_stamp: "Formatted at 2025-09-04 19:59:01 on dist-test-slave-tjsx"
I20250904 19:59:01.292520 5503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "47edf36cc269417ba960d1c1955afde1"
format_stamp: "Formatted at 2025-09-04 19:59:01 on dist-test-slave-tjsx"
I20250904 19:59:01.299321 5503 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250904 19:59:01.304855 5519 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:01.305837 5503 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250904 19:59:01.306128 5503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "47edf36cc269417ba960d1c1955afde1"
format_stamp: "Formatted at 2025-09-04 19:59:01 on dist-test-slave-tjsx"
I20250904 19:59:01.306412 5503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:01.362062 5503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:01.363503 5503 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:01.363911 5503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:01.366637 5503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:01.370756 5503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:59:01.370941 5503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:01.371192 5503 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:59:01.371340 5503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:01.520170 5503 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36843
I20250904 19:59:01.520344 5631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:36843 every 8 connection(s)
I20250904 19:59:01.522634 5503 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 19:59:01.531793 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 5503
I20250904 19:59:01.532272 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250904 19:59:01.539157 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38495
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:59:01.543977 5632 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:01.544366 5632 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:01.545327 5632 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:01.547576 5444 ts_manager.cc:194] Registered new tserver with Master: 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843)
I20250904 19:59:01.549983 5444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:32981
W20250904 19:59:01.827291 5636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:01.827772 5636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:01.828253 5636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:01.858635 5636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:01.859457 5636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 19:59:01.893122 5636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:01.894313 5636 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:01.896085 5636 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:01.907748 5642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:02.553622 5632 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
W20250904 19:59:01.908782 5643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:02.991250 5644 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1083 milliseconds
W20250904 19:59:02.992048 5645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:02.992086 5636 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:02.996484 5636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:02.998965 5636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:03.000387 5636 hybrid_clock.cc:648] HybridClock initialized: now 1757015943000352 us; error 34 us; skew 500 ppm
I20250904 19:59:03.001180 5636 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:03.007547 5636 webserver.cc:480] Webserver started at http://127.2.182.2:42293/ using document root <none> and password file <none>
I20250904 19:59:03.008405 5636 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:03.008649 5636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:03.009066 5636 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:59:03.013237 5636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2e799fbd30c84f32af95370d7caf8fcc"
format_stamp: "Formatted at 2025-09-04 19:59:03 on dist-test-slave-tjsx"
I20250904 19:59:03.014257 5636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2e799fbd30c84f32af95370d7caf8fcc"
format_stamp: "Formatted at 2025-09-04 19:59:03 on dist-test-slave-tjsx"
I20250904 19:59:03.020668 5636 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250904 19:59:03.025894 5652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:03.026832 5636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250904 19:59:03.027128 5636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2e799fbd30c84f32af95370d7caf8fcc"
format_stamp: "Formatted at 2025-09-04 19:59:03 on dist-test-slave-tjsx"
I20250904 19:59:03.027544 5636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:03.090293 5636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:03.091938 5636 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:03.092415 5636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:03.094935 5636 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:03.098852 5636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:59:03.099064 5636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:03.099308 5636 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:59:03.099478 5636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:03.224684 5636 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:37583
I20250904 19:59:03.224784 5764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:37583 every 8 connection(s)
I20250904 19:59:03.227135 5636 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250904 19:59:03.236397 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 5636
I20250904 19:59:03.236753 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250904 19:59:03.242765 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38495
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:59:03.246723 5765 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:03.247097 5765 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:03.248008 5765 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:03.249871 5444 ts_manager.cc:194] Registered new tserver with Master: 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583)
I20250904 19:59:03.251060 5444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:60645
W20250904 19:59:03.530813 5769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:03.531417 5769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:03.532158 5769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:03.564146 5769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:03.564920 5769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 19:59:03.597793 5769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:03.598986 5769 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:03.600419 5769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:03.611488 5775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:04.253722 5765 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
W20250904 19:59:03.612437 5776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:04.707347 5778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:04.709918 5777 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1093 milliseconds
I20250904 19:59:04.710043 5769 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:04.711171 5769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:04.713521 5769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:04.714910 5769 hybrid_clock.cc:648] HybridClock initialized: now 1757015944714868 us; error 42 us; skew 500 ppm
I20250904 19:59:04.715649 5769 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:04.721395 5769 webserver.cc:480] Webserver started at http://127.2.182.3:40191/ using document root <none> and password file <none>
I20250904 19:59:04.722216 5769 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:04.722411 5769 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:04.722874 5769 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:59:04.727260 5769 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d52eee06fc004fe2bb9d1d3619265627"
format_stamp: "Formatted at 2025-09-04 19:59:04 on dist-test-slave-tjsx"
I20250904 19:59:04.728330 5769 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d52eee06fc004fe2bb9d1d3619265627"
format_stamp: "Formatted at 2025-09-04 19:59:04 on dist-test-slave-tjsx"
I20250904 19:59:04.735013 5769 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250904 19:59:04.740254 5785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:04.741179 5769 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250904 19:59:04.741492 5769 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d52eee06fc004fe2bb9d1d3619265627"
format_stamp: "Formatted at 2025-09-04 19:59:04 on dist-test-slave-tjsx"
I20250904 19:59:04.741820 5769 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:04.808502 5769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:04.809940 5769 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:04.810374 5769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:04.812791 5769 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:04.816728 5769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 19:59:04.816923 5769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:04.817178 5769 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 19:59:04.817322 5769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:04.939494 5769 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:38161
I20250904 19:59:04.939616 5897 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:38161 every 8 connection(s)
I20250904 19:59:04.941846 5769 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250904 19:59:04.950336 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 5769
I20250904 19:59:04.950744 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250904 19:59:04.965807 5898 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:04.966199 5898 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:04.967362 5898 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:04.969445 5443 ts_manager.cc:194] Registered new tserver with Master: d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161)
I20250904 19:59:04.970795 5443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:58217
I20250904 19:59:04.982818 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:59:05.011533 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:59:05.011833 2776 test_util.cc:276] Using random seed: 658793061
I20250904 19:59:05.048889 5443 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37438:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 19:59:05.086105 5567 tablet_service.cc:1468] Processing CreateTablet for tablet fffc60dadf624310ac4cb8f96f498653 (DEFAULT_TABLE table=TestTable [id=7c944ca4876547d4bf3bf9583ae125f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:59:05.087496 5567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fffc60dadf624310ac4cb8f96f498653. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:59:05.104591 5918 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Bootstrap starting.
I20250904 19:59:05.109983 5918 tablet_bootstrap.cc:654] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Neither blocks nor log segments found. Creating new log.
I20250904 19:59:05.111634 5918 log.cc:826] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:05.115880 5918 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: No bootstrap required, opened a new log
I20250904 19:59:05.116262 5918 ts_tablet_manager.cc:1397] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250904 19:59:05.133029 5918 raft_consensus.cc:357] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:05.133569 5918 raft_consensus.cc:383] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:59:05.133798 5918 raft_consensus.cc:738] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47edf36cc269417ba960d1c1955afde1, State: Initialized, Role: FOLLOWER
I20250904 19:59:05.134433 5918 consensus_queue.cc:260] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:05.134964 5918 raft_consensus.cc:397] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:05.135243 5918 raft_consensus.cc:491] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:05.135522 5918 raft_consensus.cc:3058] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:59:05.139645 5918 raft_consensus.cc:513] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:05.140285 5918 leader_election.cc:304] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47edf36cc269417ba960d1c1955afde1; no voters:
I20250904 19:59:05.141911 5918 leader_election.cc:290] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:59:05.142241 5920 raft_consensus.cc:2802] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:59:05.144155 5920 raft_consensus.cc:695] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 1 LEADER]: Becoming Leader. State: Replica: 47edf36cc269417ba960d1c1955afde1, State: Running, Role: LEADER
I20250904 19:59:05.144965 5920 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:05.145408 5918 ts_tablet_manager.cc:1428] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Time spent starting tablet: real 0.029s user 0.023s sys 0.006s
I20250904 19:59:05.156536 5443 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47edf36cc269417ba960d1c1955afde1 (127.2.182.1). New cstate: current_term: 1 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } health_report { overall_health: HEALTHY } } }
I20250904 19:59:05.372478 2776 test_util.cc:276] Using random seed: 659153696
I20250904 19:59:05.391330 5438 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37448:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 19:59:05.418076 5833 tablet_service.cc:1468] Processing CreateTablet for tablet 8d6c85b5a6e04a1bb62574c5e27d4355 (DEFAULT_TABLE table=TestTable1 [id=aeee423bd98d4ffc956f751d973ae634]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:59:05.419485 5833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d6c85b5a6e04a1bb62574c5e27d4355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:59:05.437616 5939 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap starting.
I20250904 19:59:05.442839 5939 tablet_bootstrap.cc:654] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Neither blocks nor log segments found. Creating new log.
I20250904 19:59:05.444386 5939 log.cc:826] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:05.448222 5939 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: No bootstrap required, opened a new log
I20250904 19:59:05.448596 5939 ts_tablet_manager.cc:1397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250904 19:59:05.464658 5939 raft_consensus.cc:357] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } }
I20250904 19:59:05.465194 5939 raft_consensus.cc:383] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:59:05.465422 5939 raft_consensus.cc:738] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52eee06fc004fe2bb9d1d3619265627, State: Initialized, Role: FOLLOWER
I20250904 19:59:05.466009 5939 consensus_queue.cc:260] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } }
I20250904 19:59:05.466496 5939 raft_consensus.cc:397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:05.466759 5939 raft_consensus.cc:491] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:05.467029 5939 raft_consensus.cc:3058] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:59:05.470758 5939 raft_consensus.cc:513] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } }
I20250904 19:59:05.471352 5939 leader_election.cc:304] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d52eee06fc004fe2bb9d1d3619265627; no voters:
I20250904 19:59:05.472661 5939 leader_election.cc:290] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:59:05.472990 5941 raft_consensus.cc:2802] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:59:05.475045 5941 raft_consensus.cc:695] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 1 LEADER]: Becoming Leader. State: Replica: d52eee06fc004fe2bb9d1d3619265627, State: Running, Role: LEADER
I20250904 19:59:05.476019 5898 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
I20250904 19:59:05.475822 5941 consensus_queue.cc:237] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } }
I20250904 19:59:05.477691 5939 ts_tablet_manager.cc:1428] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Time spent starting tablet: real 0.029s user 0.022s sys 0.008s
I20250904 19:59:05.485659 5438 catalog_manager.cc:5582] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 reported cstate change: term changed from 0 to 1, leader changed from <none> to d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "d52eee06fc004fe2bb9d1d3619265627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } health_report { overall_health: HEALTHY } } }
I20250904 19:59:05.699066 2776 test_util.cc:276] Using random seed: 659480172
I20250904 19:59:05.718415 5440 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37452:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250904 19:59:05.745004 5700 tablet_service.cc:1468] Processing CreateTablet for tablet 078bbd461f134ecb9712bedaf958058a (DEFAULT_TABLE table=TestTable2 [id=e097083716534c03aefc656d01943d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 19:59:05.746330 5700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078bbd461f134ecb9712bedaf958058a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:59:05.763933 5961 tablet_bootstrap.cc:492] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap starting.
I20250904 19:59:05.769296 5961 tablet_bootstrap.cc:654] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Neither blocks nor log segments found. Creating new log.
I20250904 19:59:05.770859 5961 log.cc:826] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:05.774717 5961 tablet_bootstrap.cc:492] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: No bootstrap required, opened a new log
I20250904 19:59:05.775061 5961 ts_tablet_manager.cc:1397] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20250904 19:59:05.792001 5961 raft_consensus.cc:357] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:05.792491 5961 raft_consensus.cc:383] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:59:05.792677 5961 raft_consensus.cc:738] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Initialized, Role: FOLLOWER
I20250904 19:59:05.793282 5961 consensus_queue.cc:260] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:05.793785 5961 raft_consensus.cc:397] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:05.794011 5961 raft_consensus.cc:491] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:05.794250 5961 raft_consensus.cc:3058] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:59:05.798070 5961 raft_consensus.cc:513] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:05.798662 5961 leader_election.cc:304] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e799fbd30c84f32af95370d7caf8fcc; no voters:
I20250904 19:59:05.800271 5961 leader_election.cc:290] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:59:05.800617 5963 raft_consensus.cc:2802] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:59:05.803526 5963 raft_consensus.cc:695] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 1 LEADER]: Becoming Leader. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Running, Role: LEADER
I20250904 19:59:05.803916 5961 ts_tablet_manager.cc:1428] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Time spent starting tablet: real 0.029s user 0.016s sys 0.012s
I20250904 19:59:05.804279 5963 consensus_queue.cc:237] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:05.815235 5440 catalog_manager.cc:5582] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2). New cstate: current_term: 1 leader_uuid: "2e799fbd30c84f32af95370d7caf8fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } health_report { overall_health: HEALTHY } } }
I20250904 19:59:06.024981 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 5411
W20250904 19:59:06.171948 5632 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38495: connect: Connection refused (error 111)
W20250904 19:59:06.504295 5898 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38495: connect: Connection refused (error 111)
W20250904 19:59:06.829679 5765 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38495: connect: Connection refused (error 111)
W20250904 19:59:09.471562 5628 debug-util.cc:398] Leaking SignalData structure 0x7b08000c2960 after lost signal to thread 5504
W20250904 19:59:09.472550 5628 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce9e0 after lost signal to thread 5631
I20250904 19:59:10.712271 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 5503
I20250904 19:59:10.731230 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 5636
I20250904 19:59:10.750671 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 5769
I20250904 19:59:10.770583 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38495
--webserver_interface=127.2.182.62
--webserver_port=43493
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38495 with env {}
W20250904 19:59:11.054929 6039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:11.055500 6039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:11.055949 6039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:11.086185 6039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:59:11.086486 6039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:11.086767 6039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:59:11.086995 6039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:59:11.120664 6039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38495
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=43493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:11.121902 6039 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:11.123411 6039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:11.132826 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:12.536207 6044 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 6039
W20250904 19:59:12.652561 6039 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.519s user 0.635s sys 0.879s
W20250904 19:59:11.133980 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:12.652930 6039 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.520s user 0.636s sys 0.879s
W20250904 19:59:12.655055 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:12.657400 6047 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1519 milliseconds
I20250904 19:59:12.657469 6039 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:12.658568 6039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:12.660844 6039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:12.662170 6039 hybrid_clock.cc:648] HybridClock initialized: now 1757015952662130 us; error 39 us; skew 500 ppm
I20250904 19:59:12.662955 6039 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:12.668419 6039 webserver.cc:480] Webserver started at http://127.2.182.62:43493/ using document root <none> and password file <none>
I20250904 19:59:12.669268 6039 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:12.669482 6039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:12.676635 6039 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250904 19:59:12.680727 6055 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:12.681644 6039 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250904 19:59:12.681933 6039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "116b311348744b64b8e53eec1aad5435"
format_stamp: "Formatted at 2025-09-04 19:58:59 on dist-test-slave-tjsx"
I20250904 19:59:12.683766 6039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:12.753165 6039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:12.754560 6039 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:12.754993 6039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:12.819454 6039 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38495
I20250904 19:59:12.819547 6106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:38495 every 8 connection(s)
I20250904 19:59:12.822068 6039 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:59:12.829891 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 6039
I20250904 19:59:12.831259 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:36843
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:38495
--webserver_port=35221
--webserver_interface=127.2.182.1
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:59:12.831703 6107 sys_catalog.cc:263] Verifying existing consensus state
I20250904 19:59:12.836091 6107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap starting.
I20250904 19:59:12.845328 6107 log.cc:826] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:12.891930 6107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:12.892696 6107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap complete.
I20250904 19:59:12.912699 6107 raft_consensus.cc:357] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:12.914783 6107 raft_consensus.cc:738] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116b311348744b64b8e53eec1aad5435, State: Initialized, Role: FOLLOWER
I20250904 19:59:12.915516 6107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:12.916033 6107 raft_consensus.cc:397] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:12.916291 6107 raft_consensus.cc:491] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:12.916591 6107 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 2 FOLLOWER]: Advancing to term 3
I20250904 19:59:12.921658 6107 raft_consensus.cc:513] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:12.922248 6107 leader_election.cc:304] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116b311348744b64b8e53eec1aad5435; no voters:
I20250904 19:59:12.924093 6107 leader_election.cc:290] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250904 19:59:12.924471 6111 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 3 FOLLOWER]: Leader election won for term 3
I20250904 19:59:12.927309 6111 raft_consensus.cc:695] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 3 LEADER]: Becoming Leader. State: Replica: 116b311348744b64b8e53eec1aad5435, State: Running, Role: LEADER
I20250904 19:59:12.928421 6107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:59:12.928128 6111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:12.936803 6113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116b311348744b64b8e53eec1aad5435. Latest consensus state: current_term: 3 leader_uuid: "116b311348744b64b8e53eec1aad5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } } }
I20250904 19:59:12.937889 6112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "116b311348744b64b8e53eec1aad5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } } }
I20250904 19:59:12.938534 6112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: This master's current role is: LEADER
I20250904 19:59:12.940589 6113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: This master's current role is: LEADER
I20250904 19:59:12.951022 6118 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:59:12.962354 6118 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=71071a0dabcb4e2187ea57b1e2d74388]
I20250904 19:59:12.963936 6118 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d515a202718641119728c88e7adedc4c]
I20250904 19:59:12.965495 6118 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=e097083716534c03aefc656d01943d7c]
I20250904 19:59:12.972630 6118 tablet_loader.cc:96] loaded metadata for tablet 078bbd461f134ecb9712bedaf958058a (table TestTable2 [id=e097083716534c03aefc656d01943d7c])
I20250904 19:59:12.973875 6118 tablet_loader.cc:96] loaded metadata for tablet 8d6c85b5a6e04a1bb62574c5e27d4355 (table TestTable1 [id=71071a0dabcb4e2187ea57b1e2d74388])
I20250904 19:59:12.975066 6118 tablet_loader.cc:96] loaded metadata for tablet fffc60dadf624310ac4cb8f96f498653 (table TestTable [id=d515a202718641119728c88e7adedc4c])
I20250904 19:59:12.976346 6118 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:59:12.981041 6118 catalog_manager.cc:1261] Loaded cluster ID: 0fe8823e3bc14e35badbe62ce3e59c2a
I20250904 19:59:12.981325 6118 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:59:12.988919 6118 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:59:12.993826 6118 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Loaded TSK: 0
I20250904 19:59:12.995424 6118 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250904 19:59:13.145339 6109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:13.145838 6109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:13.146318 6109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:13.176968 6109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:13.177804 6109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 19:59:13.212069 6109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:36843
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=35221
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:13.213295 6109 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:13.214953 6109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:13.226545 6134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:13.227449 6135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:14.559981 6137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:14.564152 6136 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1335 milliseconds
W20250904 19:59:14.564626 6109 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.337s user 0.589s sys 0.737s
W20250904 19:59:14.564952 6109 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.337s user 0.589s sys 0.737s
I20250904 19:59:14.565204 6109 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:14.566576 6109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:14.571219 6109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:14.572695 6109 hybrid_clock.cc:648] HybridClock initialized: now 1757015954572617 us; error 83 us; skew 500 ppm
I20250904 19:59:14.573734 6109 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:14.587049 6109 webserver.cc:480] Webserver started at http://127.2.182.1:35221/ using document root <none> and password file <none>
I20250904 19:59:14.588201 6109 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:14.588486 6109 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:14.599041 6109 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250904 19:59:14.606580 6144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:14.607869 6109 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250904 19:59:14.608237 6109 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "47edf36cc269417ba960d1c1955afde1"
format_stamp: "Formatted at 2025-09-04 19:59:01 on dist-test-slave-tjsx"
I20250904 19:59:14.610896 6109 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:14.692732 6109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:14.694168 6109 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:14.694561 6109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:14.697299 6109 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:14.703418 6151 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 19:59:14.710477 6109 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 19:59:14.710665 6109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250904 19:59:14.710935 6109 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 19:59:14.715334 6109 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 19:59:14.715571 6109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250904 19:59:14.715896 6151 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Bootstrap starting.
I20250904 19:59:14.786191 6151 log.cc:826] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:14.902257 6109 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36843
I20250904 19:59:14.902523 6258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:36843 every 8 connection(s)
I20250904 19:59:14.905601 6109 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 19:59:14.913256 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 6109
I20250904 19:59:14.915216 6151 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:14.915153 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:37583
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:38495
--webserver_port=42293
--webserver_interface=127.2.182.2
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:59:14.916225 6151 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Bootstrap complete.
I20250904 19:59:14.917312 6151 ts_tablet_manager.cc:1397] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Time spent bootstrapping tablet: real 0.202s user 0.160s sys 0.037s
I20250904 19:59:14.934542 6151 raft_consensus.cc:357] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:14.937443 6151 raft_consensus.cc:738] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47edf36cc269417ba960d1c1955afde1, State: Initialized, Role: FOLLOWER
I20250904 19:59:14.938422 6151 consensus_queue.cc:260] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:14.939088 6151 raft_consensus.cc:397] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:14.939464 6151 raft_consensus.cc:491] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:14.939865 6151 raft_consensus.cc:3058] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 1 FOLLOWER]: Advancing to term 2
I20250904 19:59:14.946630 6259 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:14.947096 6259 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:14.947606 6151 raft_consensus.cc:513] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:14.948199 6259 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:14.948393 6151 leader_election.cc:304] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47edf36cc269417ba960d1c1955afde1; no voters:
I20250904 19:59:14.952489 6072 ts_manager.cc:194] Registered new tserver with Master: 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843)
I20250904 19:59:14.959698 6072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:57763
I20250904 19:59:14.966122 6151 leader_election.cc:290] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250904 19:59:14.966971 6264 raft_consensus.cc:2802] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 19:59:14.974800 6264 raft_consensus.cc:695] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEADER]: Becoming Leader. State: Replica: 47edf36cc269417ba960d1c1955afde1, State: Running, Role: LEADER
I20250904 19:59:14.975641 6264 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:14.978540 6259 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
I20250904 19:59:14.979938 6151 ts_tablet_manager.cc:1428] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Time spent starting tablet: real 0.062s user 0.041s sys 0.017s
I20250904 19:59:14.988870 6072 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 reported cstate change: term changed from 0 to 2, leader changed from <none> to 47edf36cc269417ba960d1c1955afde1 (127.2.182.1), VOTER 47edf36cc269417ba960d1c1955afde1 (127.2.182.1) added. New cstate: current_term: 2 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } health_report { overall_health: HEALTHY } } }
W20250904 19:59:15.030211 6072 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fffc60dadf624310ac4cb8f96f498653 with cas_config_opid_index -1: no extra replica candidate found for tablet fffc60dadf624310ac4cb8f96f498653 (table TestTable [id=d515a202718641119728c88e7adedc4c]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250904 19:59:15.241672 6263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:15.242139 6263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:15.242624 6263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:15.274184 6263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:15.275022 6263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 19:59:15.309242 6263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:37583
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=42293
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:15.310475 6263 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:15.311964 6263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:15.323465 6278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:15.325726 6279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:16.578306 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:16.579982 6280 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1255 milliseconds
W20250904 19:59:16.580982 6263 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.257s user 0.441s sys 0.808s
W20250904 19:59:16.581333 6263 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.258s user 0.442s sys 0.809s
I20250904 19:59:16.581588 6263 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:16.582718 6263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:16.587332 6263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:16.588733 6263 hybrid_clock.cc:648] HybridClock initialized: now 1757015956588685 us; error 47 us; skew 500 ppm
I20250904 19:59:16.589474 6263 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:16.596161 6263 webserver.cc:480] Webserver started at http://127.2.182.2:42293/ using document root <none> and password file <none>
I20250904 19:59:16.597016 6263 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:16.597219 6263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:16.604753 6263 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250904 19:59:16.609730 6289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:16.610803 6263 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250904 19:59:16.611094 6263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2e799fbd30c84f32af95370d7caf8fcc"
format_stamp: "Formatted at 2025-09-04 19:59:03 on dist-test-slave-tjsx"
I20250904 19:59:16.612972 6263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:16.668054 6263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:16.669641 6263 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:16.670037 6263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:16.672753 6263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:16.677956 6296 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 19:59:16.684891 6263 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 19:59:16.685096 6263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.002s sys 0.000s
I20250904 19:59:16.685338 6263 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 19:59:16.689545 6263 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 19:59:16.689728 6263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20250904 19:59:16.690044 6296 tablet_bootstrap.cc:492] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap starting.
I20250904 19:59:16.743398 6296 log.cc:826] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:16.835865 6296 tablet_bootstrap.cc:492] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:16.836853 6296 tablet_bootstrap.cc:492] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap complete.
I20250904 19:59:16.838469 6296 ts_tablet_manager.cc:1397] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Time spent bootstrapping tablet: real 0.149s user 0.110s sys 0.036s
I20250904 19:59:16.848067 6263 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:37583
I20250904 19:59:16.848233 6403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:37583 every 8 connection(s)
I20250904 19:59:16.850386 6263 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250904 19:59:16.856369 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 6263
I20250904 19:59:16.858368 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:38161
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:38495
--webserver_port=40191
--webserver_interface=127.2.182.3
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:59:16.857404 6296 raft_consensus.cc:357] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:16.859510 6296 raft_consensus.cc:738] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Initialized, Role: FOLLOWER
I20250904 19:59:16.860280 6296 consensus_queue.cc:260] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:16.860772 6296 raft_consensus.cc:397] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:16.861020 6296 raft_consensus.cc:491] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:16.861279 6296 raft_consensus.cc:3058] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 1 FOLLOWER]: Advancing to term 2
I20250904 19:59:16.867453 6296 raft_consensus.cc:513] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:16.868144 6296 leader_election.cc:304] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e799fbd30c84f32af95370d7caf8fcc; no voters:
I20250904 19:59:16.869892 6296 leader_election.cc:290] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 2 election: Requested vote from peers
I20250904 19:59:16.870733 6409 raft_consensus.cc:2802] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Leader election won for term 2
I20250904 19:59:16.874646 6404 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:16.875175 6404 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:16.876394 6404 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:16.876427 6409 raft_consensus.cc:695] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEADER]: Becoming Leader. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Running, Role: LEADER
I20250904 19:59:16.876740 6296 ts_tablet_manager.cc:1428] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Time spent starting tablet: real 0.038s user 0.031s sys 0.004s
I20250904 19:59:16.877261 6409 consensus_queue.cc:237] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:16.885919 6072 ts_manager.cc:194] Registered new tserver with Master: 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583)
I20250904 19:59:16.886869 6072 catalog_manager.cc:5582] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "2e799fbd30c84f32af95370d7caf8fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } health_report { overall_health: HEALTHY } } }
I20250904 19:59:16.896181 6072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:33011
I20250904 19:59:16.898881 6404 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
W20250904 19:59:17.162473 6408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:17.162981 6408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:17.163471 6408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:17.193511 6408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:17.194320 6408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 19:59:17.227806 6408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:38161
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=40191
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:17.229049 6408 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:17.230593 6408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:17.241081 6423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:17.376613 6214 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
I20250904 19:59:17.412364 6433 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } } }
I20250904 19:59:17.426168 6058 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fffc60dadf624310ac4cb8f96f498653 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250904 19:59:17.432118 6072 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 reported cstate change: config changed from index -1 to 10, NON_VOTER 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) added. New cstate: current_term: 2 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250904 19:59:17.435503 6148 consensus_peers.cc:489] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 -> Peer 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583): Couldn't send request to peer 2e799fbd30c84f32af95370d7caf8fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fffc60dadf624310ac4cb8f96f498653. This is attempt 1: this message will repeat every 5th retry.
I20250904 19:59:17.925928 6439 ts_tablet_manager.cc:927] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Initiating tablet copy from peer 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843)
I20250904 19:59:17.947722 6439 tablet_copy_client.cc:323] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.1:36843
I20250904 19:59:17.969614 6234 tablet_copy_service.cc:140] P 47edf36cc269417ba960d1c1955afde1: Received BeginTabletCopySession request for tablet fffc60dadf624310ac4cb8f96f498653 from peer 2e799fbd30c84f32af95370d7caf8fcc ({username='slave'} at 127.2.182.2:35007)
I20250904 19:59:17.970238 6234 tablet_copy_service.cc:161] P 47edf36cc269417ba960d1c1955afde1: Beginning new tablet copy session on tablet fffc60dadf624310ac4cb8f96f498653 from peer 2e799fbd30c84f32af95370d7caf8fcc at {username='slave'} at 127.2.182.2:35007: session id = 2e799fbd30c84f32af95370d7caf8fcc-fffc60dadf624310ac4cb8f96f498653
I20250904 19:59:17.977737 6234 tablet_copy_source_session.cc:215] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Tablet Copy: opened 0 blocks and 1 log segments
I20250904 19:59:17.990202 6439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fffc60dadf624310ac4cb8f96f498653. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:59:18.038820 6439 tablet_copy_client.cc:806] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: tablet copy: Starting download of 0 data blocks...
I20250904 19:59:18.039688 6439 tablet_copy_client.cc:670] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: tablet copy: Starting download of 1 WAL segments...
I20250904 19:59:18.046782 6439 tablet_copy_client.cc:538] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250904 19:59:18.067466 6439 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap starting.
I20250904 19:59:18.249667 6439 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:18.250658 6439 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap complete.
I20250904 19:59:18.251376 6439 ts_tablet_manager.cc:1397] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Time spent bootstrapping tablet: real 0.184s user 0.127s sys 0.024s
I20250904 19:59:18.255066 6439 raft_consensus.cc:357] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
I20250904 19:59:18.255771 6439 raft_consensus.cc:738] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Initialized, Role: LEARNER
I20250904 19:59:18.256547 6439 consensus_queue.cc:260] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
I20250904 19:59:18.265166 6439 ts_tablet_manager.cc:1428] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20250904 19:59:18.266882 6234 tablet_copy_service.cc:342] P 47edf36cc269417ba960d1c1955afde1: Request end of tablet copy session 2e799fbd30c84f32af95370d7caf8fcc-fffc60dadf624310ac4cb8f96f498653 received from {username='slave'} at 127.2.182.2:35007
I20250904 19:59:18.267376 6234 tablet_copy_service.cc:434] P 47edf36cc269417ba960d1c1955afde1: ending tablet copy session 2e799fbd30c84f32af95370d7caf8fcc-fffc60dadf624310ac4cb8f96f498653 on tablet fffc60dadf624310ac4cb8f96f498653 with peer 2e799fbd30c84f32af95370d7caf8fcc
I20250904 19:59:18.339121 6359 raft_consensus.cc:1215] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250904 19:59:18.764293 6433 raft_consensus.cc:1062] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: attempting to promote NON_VOTER 2e799fbd30c84f32af95370d7caf8fcc to VOTER
I20250904 19:59:18.766115 6433 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:18.775139 6359 raft_consensus.cc:1273] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Refusing update from remote peer 47edf36cc269417ba960d1c1955afde1: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250904 19:59:18.777621 6446 consensus_queue.cc:1035] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250904 19:59:18.795840 6433 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
I20250904 19:59:18.802892 6358 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
I20250904 19:59:18.815698 6072 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc reported cstate change: config changed from index 10 to 11, 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
W20250904 19:59:18.644023 6422 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 6408
W20250904 19:59:18.833316 6422 kernel_stack_watchdog.cc:198] Thread 6408 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 19:59:17.242563 6424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:18.835305 6408 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.593s user 0.640s sys 0.900s
W20250904 19:59:18.835701 6408 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.594s user 0.640s sys 0.900s
W20250904 19:59:18.836004 6426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:18.840525 6425 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1592 milliseconds
I20250904 19:59:18.841101 6408 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:18.842577 6408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:18.846088 6408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:18.847751 6408 hybrid_clock.cc:648] HybridClock initialized: now 1757015958847664 us; error 98 us; skew 500 ppm
I20250904 19:59:18.850064 6408 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:18.875588 6408 webserver.cc:480] Webserver started at http://127.2.182.3:40191/ using document root <none> and password file <none>
I20250904 19:59:18.876997 6408 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:18.877312 6408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:18.890157 6408 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.005s sys 0.005s
I20250904 19:59:18.896486 6457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:18.897840 6408 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20250904 19:59:18.898204 6408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d52eee06fc004fe2bb9d1d3619265627"
format_stamp: "Formatted at 2025-09-04 19:59:04 on dist-test-slave-tjsx"
I20250904 19:59:18.900871 6408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:18.957768 6408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:18.959323 6408 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:18.959841 6408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:18.962548 6408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:18.968521 6464 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 19:59:18.976207 6408 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 19:59:18.976475 6408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250904 19:59:18.976770 6408 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 19:59:18.981635 6408 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 19:59:18.981834 6408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250904 19:59:18.982304 6464 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap starting.
I20250904 19:59:19.041006 6464 log.cc:826] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:19.161255 6408 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:38161
I20250904 19:59:19.161473 6571 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:38161 every 8 connection(s)
I20250904 19:59:19.165015 6408 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250904 19:59:19.173002 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 6408
I20250904 19:59:19.176654 6464 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:19.178246 6464 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap complete.
I20250904 19:59:19.180423 6464 ts_tablet_manager.cc:1397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Time spent bootstrapping tablet: real 0.198s user 0.165s sys 0.028s
I20250904 19:59:19.199644 6572 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:19.200309 6572 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:19.198539 6464 raft_consensus.cc:357] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } }
I20250904 19:59:19.201764 6464 raft_consensus.cc:738] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52eee06fc004fe2bb9d1d3619265627, State: Initialized, Role: FOLLOWER
I20250904 19:59:19.201881 6572 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:19.203510 6464 consensus_queue.cc:260] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } }
I20250904 19:59:19.204200 6464 raft_consensus.cc:397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:19.204671 6464 raft_consensus.cc:491] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:19.205163 6464 raft_consensus.cc:3058] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 1 FOLLOWER]: Advancing to term 2
I20250904 19:59:19.213244 6071 ts_manager.cc:194] Registered new tserver with Master: d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161)
I20250904 19:59:19.214440 6464 raft_consensus.cc:513] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } }
I20250904 19:59:19.215301 6464 leader_election.cc:304] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d52eee06fc004fe2bb9d1d3619265627; no voters:
I20250904 19:59:19.216001 6071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:49005
I20250904 19:59:19.217407 6464 leader_election.cc:290] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250904 19:59:19.218369 6578 raft_consensus.cc:2802] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 19:59:19.221783 6214 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: NON_VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: true } }
I20250904 19:59:19.225688 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:59:19.226624 6578 raft_consensus.cc:695] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEADER]: Becoming Leader. State: Replica: d52eee06fc004fe2bb9d1d3619265627, State: Running, Role: LEADER
I20250904 19:59:19.227572 6578 consensus_queue.cc:237] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } }
I20250904 19:59:19.228482 6464 ts_tablet_manager.cc:1428] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Time spent starting tablet: real 0.048s user 0.035s sys 0.007s
I20250904 19:59:19.229441 6572 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
I20250904 19:59:19.232115 6358 raft_consensus.cc:1273] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Refusing update from remote peer 47edf36cc269417ba960d1c1955afde1: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250904 19:59:19.233827 6446 consensus_queue.cc:1035] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250904 19:59:19.234870 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:59:19.240973 6433 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: NON_VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: true } } }
I20250904 19:59:19.244688 6071 catalog_manager.cc:5582] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 reported cstate change: term changed from 0 to 2, leader changed from <none> to d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3), VOTER d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) added. New cstate: current_term: 2 leader_uuid: "d52eee06fc004fe2bb9d1d3619265627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } health_report { overall_health: HEALTHY } } }
W20250904 19:59:19.244659 2776 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250904 19:59:19.247829 6358 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: NON_VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: true } } }
I20250904 19:59:19.251138 6058 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fffc60dadf624310ac4cb8f96f498653 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250904 19:59:19.255856 6072 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 reported cstate change: config changed from index 11 to 12, NON_VOTER d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) added. New cstate: current_term: 2 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: NON_VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250904 19:59:19.263183 6146 consensus_peers.cc:489] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 -> Peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Couldn't send request to peer d52eee06fc004fe2bb9d1d3619265627. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fffc60dadf624310ac4cb8f96f498653. This is attempt 1: this message will repeat every 5th retry.
I20250904 19:59:19.278344 6527 consensus_queue.cc:237] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } }
I20250904 19:59:19.282074 6578 raft_consensus.cc:2953] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 47edf36cc269417ba960d1c1955afde1 (127.2.182.1) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } } }
I20250904 19:59:19.290200 6057 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d6c85b5a6e04a1bb62574c5e27d4355 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250904 19:59:19.293179 6070 catalog_manager.cc:5582] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 reported cstate change: config changed from index -1 to 9, NON_VOTER 47edf36cc269417ba960d1c1955afde1 (127.2.182.1) added. New cstate: current_term: 2 leader_uuid: "d52eee06fc004fe2bb9d1d3619265627" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250904 19:59:19.294669 6460 consensus_peers.cc:489] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 -> Peer 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843): Couldn't send request to peer 47edf36cc269417ba960d1c1955afde1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d6c85b5a6e04a1bb62574c5e27d4355. This is attempt 1: this message will repeat every 5th retry.
I20250904 19:59:19.304857 6527 consensus_queue.cc:237] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
W20250904 19:59:19.310576 6460 consensus_peers.cc:489] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 -> Peer 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843): Couldn't send request to peer 47edf36cc269417ba960d1c1955afde1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d6c85b5a6e04a1bb62574c5e27d4355. This is attempt 1: this message will repeat every 5th retry.
I20250904 19:59:19.308840 6579 raft_consensus.cc:2953] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } } }
I20250904 19:59:19.317652 6057 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8d6c85b5a6e04a1bb62574c5e27d4355 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250904 19:59:19.319464 6461 consensus_peers.cc:489] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 -> Peer 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583): Couldn't send request to peer 2e799fbd30c84f32af95370d7caf8fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d6c85b5a6e04a1bb62574c5e27d4355. This is attempt 1: this message will repeat every 5th retry.
I20250904 19:59:19.322099 6070 catalog_manager.cc:5582] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 reported cstate change: config changed from index 9 to 10, NON_VOTER 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) added. New cstate: current_term: 2 leader_uuid: "d52eee06fc004fe2bb9d1d3619265627" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250904 19:59:19.658496 6588 ts_tablet_manager.cc:927] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Initiating tablet copy from peer 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843)
I20250904 19:59:19.660583 6588 tablet_copy_client.cc:323] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.1:36843
I20250904 19:59:19.662216 6234 tablet_copy_service.cc:140] P 47edf36cc269417ba960d1c1955afde1: Received BeginTabletCopySession request for tablet fffc60dadf624310ac4cb8f96f498653 from peer d52eee06fc004fe2bb9d1d3619265627 ({username='slave'} at 127.2.182.3:50793)
I20250904 19:59:19.662649 6234 tablet_copy_service.cc:161] P 47edf36cc269417ba960d1c1955afde1: Beginning new tablet copy session on tablet fffc60dadf624310ac4cb8f96f498653 from peer d52eee06fc004fe2bb9d1d3619265627 at {username='slave'} at 127.2.182.3:50793: session id = d52eee06fc004fe2bb9d1d3619265627-fffc60dadf624310ac4cb8f96f498653
I20250904 19:59:19.666975 6234 tablet_copy_source_session.cc:215] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Tablet Copy: opened 0 blocks and 1 log segments
I20250904 19:59:19.670107 6588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fffc60dadf624310ac4cb8f96f498653. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:59:19.679668 6058 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fffc60dadf624310ac4cb8f96f498653 with cas_config_opid_index 10: aborting the task: latest config opid_index 12; task opid_index 10
I20250904 19:59:19.684417 6588 tablet_copy_client.cc:806] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: tablet copy: Starting download of 0 data blocks...
I20250904 19:59:19.684875 6588 tablet_copy_client.cc:670] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: tablet copy: Starting download of 1 WAL segments...
I20250904 19:59:19.689424 6588 tablet_copy_client.cc:538] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250904 19:59:19.695380 6588 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap starting.
I20250904 19:59:19.774181 6591 ts_tablet_manager.cc:927] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Initiating tablet copy from peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161)
I20250904 19:59:19.776373 6591 tablet_copy_client.cc:323] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.3:38161
I20250904 19:59:19.778378 6547 tablet_copy_service.cc:140] P d52eee06fc004fe2bb9d1d3619265627: Received BeginTabletCopySession request for tablet 8d6c85b5a6e04a1bb62574c5e27d4355 from peer 47edf36cc269417ba960d1c1955afde1 ({username='slave'} at 127.2.182.1:42283)
I20250904 19:59:19.779044 6547 tablet_copy_service.cc:161] P d52eee06fc004fe2bb9d1d3619265627: Beginning new tablet copy session on tablet 8d6c85b5a6e04a1bb62574c5e27d4355 from peer 47edf36cc269417ba960d1c1955afde1 at {username='slave'} at 127.2.182.1:42283: session id = 47edf36cc269417ba960d1c1955afde1-8d6c85b5a6e04a1bb62574c5e27d4355
I20250904 19:59:19.786352 6547 tablet_copy_source_session.cc:215] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Tablet Copy: opened 0 blocks and 1 log segments
I20250904 19:59:19.790014 6591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d6c85b5a6e04a1bb62574c5e27d4355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:59:19.795871 6588 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:19.796542 6588 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap complete.
I20250904 19:59:19.797035 6588 ts_tablet_manager.cc:1397] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Time spent bootstrapping tablet: real 0.102s user 0.087s sys 0.012s
I20250904 19:59:19.798750 6588 raft_consensus.cc:357] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: NON_VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: true } }
I20250904 19:59:19.799261 6588 raft_consensus.cc:738] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d52eee06fc004fe2bb9d1d3619265627, State: Initialized, Role: LEARNER
I20250904 19:59:19.799698 6588 consensus_queue.cc:260] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: NON_VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: true } }
I20250904 19:59:19.801255 6588 ts_tablet_manager.cc:1428] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 19:59:19.802439 6591 tablet_copy_client.cc:806] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: tablet copy: Starting download of 0 data blocks...
I20250904 19:59:19.803246 6234 tablet_copy_service.cc:342] P 47edf36cc269417ba960d1c1955afde1: Request end of tablet copy session d52eee06fc004fe2bb9d1d3619265627-fffc60dadf624310ac4cb8f96f498653 received from {username='slave'} at 127.2.182.3:50793
I20250904 19:59:19.803691 6234 tablet_copy_service.cc:434] P 47edf36cc269417ba960d1c1955afde1: ending tablet copy session d52eee06fc004fe2bb9d1d3619265627-fffc60dadf624310ac4cb8f96f498653 on tablet fffc60dadf624310ac4cb8f96f498653 with peer d52eee06fc004fe2bb9d1d3619265627
I20250904 19:59:19.803877 6591 tablet_copy_client.cc:670] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: tablet copy: Starting download of 1 WAL segments...
I20250904 19:59:19.810967 6591 tablet_copy_client.cc:538] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250904 19:59:19.817795 6591 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Bootstrap starting.
I20250904 19:59:19.911748 6594 ts_tablet_manager.cc:927] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Initiating tablet copy from peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161)
I20250904 19:59:19.913558 6594 tablet_copy_client.cc:323] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.2.182.3:38161
I20250904 19:59:19.925361 6591 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:19.926510 6591 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Bootstrap complete.
I20250904 19:59:19.927353 6591 ts_tablet_manager.cc:1397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Time spent bootstrapping tablet: real 0.110s user 0.090s sys 0.020s
I20250904 19:59:19.930131 6591 raft_consensus.cc:357] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
I20250904 19:59:19.930769 6591 raft_consensus.cc:738] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 47edf36cc269417ba960d1c1955afde1, State: Initialized, Role: LEARNER
I20250904 19:59:19.931380 6591 consensus_queue.cc:260] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
I20250904 19:59:19.933492 6591 ts_tablet_manager.cc:1428] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250904 19:59:19.935441 6547 tablet_copy_service.cc:342] P d52eee06fc004fe2bb9d1d3619265627: Request end of tablet copy session 47edf36cc269417ba960d1c1955afde1-8d6c85b5a6e04a1bb62574c5e27d4355 received from {username='slave'} at 127.2.182.1:42283
I20250904 19:59:19.935925 6547 tablet_copy_service.cc:434] P d52eee06fc004fe2bb9d1d3619265627: ending tablet copy session 47edf36cc269417ba960d1c1955afde1-8d6c85b5a6e04a1bb62574c5e27d4355 on tablet 8d6c85b5a6e04a1bb62574c5e27d4355 with peer 47edf36cc269417ba960d1c1955afde1
I20250904 19:59:19.939397 6547 tablet_copy_service.cc:140] P d52eee06fc004fe2bb9d1d3619265627: Received BeginTabletCopySession request for tablet 8d6c85b5a6e04a1bb62574c5e27d4355 from peer 2e799fbd30c84f32af95370d7caf8fcc ({username='slave'} at 127.2.182.2:46049)
I20250904 19:59:19.939941 6547 tablet_copy_service.cc:161] P d52eee06fc004fe2bb9d1d3619265627: Beginning new tablet copy session on tablet 8d6c85b5a6e04a1bb62574c5e27d4355 from peer 2e799fbd30c84f32af95370d7caf8fcc at {username='slave'} at 127.2.182.2:46049: session id = 2e799fbd30c84f32af95370d7caf8fcc-8d6c85b5a6e04a1bb62574c5e27d4355
I20250904 19:59:19.947052 6547 tablet_copy_source_session.cc:215] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Tablet Copy: opened 0 blocks and 1 log segments
I20250904 19:59:19.950381 6594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d6c85b5a6e04a1bb62574c5e27d4355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:59:19.961843 6594 tablet_copy_client.cc:806] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: tablet copy: Starting download of 0 data blocks...
I20250904 19:59:19.962320 6594 tablet_copy_client.cc:670] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: tablet copy: Starting download of 1 WAL segments...
I20250904 19:59:19.965739 6594 tablet_copy_client.cc:538] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250904 19:59:19.971235 6594 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap starting.
I20250904 19:59:20.055480 6594 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:20.056106 6594 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap complete.
I20250904 19:59:20.056584 6594 ts_tablet_manager.cc:1397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Time spent bootstrapping tablet: real 0.086s user 0.075s sys 0.008s
I20250904 19:59:20.058121 6594 raft_consensus.cc:357] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
I20250904 19:59:20.058597 6594 raft_consensus.cc:738] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Initialized, Role: LEARNER
I20250904 19:59:20.058981 6594 consensus_queue.cc:260] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: NON_VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: true } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
I20250904 19:59:20.061480 6594 ts_tablet_manager.cc:1428] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250904 19:59:20.063382 6547 tablet_copy_service.cc:342] P d52eee06fc004fe2bb9d1d3619265627: Request end of tablet copy session 2e799fbd30c84f32af95370d7caf8fcc-8d6c85b5a6e04a1bb62574c5e27d4355 received from {username='slave'} at 127.2.182.2:46049
I20250904 19:59:20.063735 6547 tablet_copy_service.cc:434] P d52eee06fc004fe2bb9d1d3619265627: ending tablet copy session 2e799fbd30c84f32af95370d7caf8fcc-8d6c85b5a6e04a1bb62574c5e27d4355 on tablet 8d6c85b5a6e04a1bb62574c5e27d4355 with peer 2e799fbd30c84f32af95370d7caf8fcc
I20250904 19:59:20.245446 6527 raft_consensus.cc:1215] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250904 19:59:20.252082 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 47edf36cc269417ba960d1c1955afde1 to finish bootstrapping
I20250904 19:59:20.267535 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2e799fbd30c84f32af95370d7caf8fcc to finish bootstrapping
I20250904 19:59:20.282392 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d52eee06fc004fe2bb9d1d3619265627 to finish bootstrapping
I20250904 19:59:20.371541 6214 raft_consensus.cc:1215] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250904 19:59:20.500263 6358 raft_consensus.cc:1215] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250904 19:59:20.589378 6339 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250904 19:59:20.594430 6194 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250904 19:59:20.596830 6507 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250904 19:59:20.760780 6601 raft_consensus.cc:1062] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: attempting to promote NON_VOTER d52eee06fc004fe2bb9d1d3619265627 to VOTER
I20250904 19:59:20.763093 6601 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:20.778612 6358 raft_consensus.cc:1273] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Refusing update from remote peer 47edf36cc269417ba960d1c1955afde1: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250904 19:59:20.778918 6527 raft_consensus.cc:1273] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEARNER]: Refusing update from remote peer 47edf36cc269417ba960d1c1955afde1: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250904 19:59:20.780857 6632 consensus_queue.cc:1035] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250904 19:59:20.783109 6632 consensus_queue.cc:1035] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250904 19:59:20.795761 6632 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } } }
I20250904 19:59:20.796303 6526 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } } }
I20250904 19:59:20.810432 6070 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 reported cstate change: config changed from index 12 to 13, d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } } }
I20250904 19:59:20.819948 6358 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } } }
I20250904 19:59:20.826931 6613 raft_consensus.cc:1062] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: attempting to promote NON_VOTER 47edf36cc269417ba960d1c1955afde1 to VOTER
I20250904 19:59:20.829548 6613 consensus_queue.cc:237] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } }
I20250904 19:59:20.852102 6214 raft_consensus.cc:1273] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 LEARNER]: Refusing update from remote peer d52eee06fc004fe2bb9d1d3619265627: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250904 19:59:20.853892 6579 consensus_queue.cc:1035] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250904 19:59:20.862610 6358 raft_consensus.cc:1273] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Refusing update from remote peer d52eee06fc004fe2bb9d1d3619265627: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250904 19:59:20.865867 6613 consensus_queue.cc:1035] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250904 19:59:20.873322 6579 raft_consensus.cc:2953] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 47edf36cc269417ba960d1c1955afde1 (127.2.182.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } } }
I20250904 19:59:20.884912 6214 raft_consensus.cc:2953] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 47edf36cc269417ba960d1c1955afde1 (127.2.182.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } } }
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
116b311348744b64b8e53eec1aad5435 | 127.2.182.62:38495 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.182.20:34539 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
2e799fbd30c84f32af95370d7caf8fcc | 127.2.182.2:37583 | HEALTHY | <none> | 1 | 0
47edf36cc269417ba960d1c1955afde1 | 127.2.182.1:36843 | HEALTHY | <none> | 1 | 0
d52eee06fc004fe2bb9d1d3619265627 | 127.2.182.3:38161 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.182.1 | experimental | 127.2.182.1:36843
local_ip_for_outbound_sockets | 127.2.182.2 | experimental | 127.2.182.2:37583
local_ip_for_outbound_sockets | 127.2.182.3 | experimental | 127.2.182.3:38161
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.2.182.1:36843
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.2.182.2:37583
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.2.182.3:38161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.182.20:34539 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
I20250904 19:59:20.915644 6612 raft_consensus.cc:1062] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: attempting to promote NON_VOTER 2e799fbd30c84f32af95370d7caf8fcc to VOTER
I20250904 19:59:20.921167 6612 consensus_queue.cc:237] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:20.928880 6359 raft_consensus.cc:2953] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 47edf36cc269417ba960d1c1955afde1 (127.2.182.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } } }
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250904 19:59:20.947927 6358 raft_consensus.cc:1273] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 LEARNER]: Refusing update from remote peer d52eee06fc004fe2bb9d1d3619265627: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250904 19:59:20.955088 6213 raft_consensus.cc:1273] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Refusing update from remote peer d52eee06fc004fe2bb9d1d3619265627: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250904 19:59:20.957451 6645 consensus_queue.cc:1035] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250904 19:59:20.962112 6645 consensus_queue.cc:1035] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250904 19:59:20.967993 6071 catalog_manager.cc:5582] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc reported cstate change: config changed from index 10 to 11, 47edf36cc269417ba960d1c1955afde1 (127.2.182.1) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d52eee06fc004fe2bb9d1d3619265627" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: NON_VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: true } } }
I20250904 19:59:20.972718 2776 log_verifier.cc:126] Checking tablet 078bbd461f134ecb9712bedaf958058a
I20250904 19:59:20.975476 6213 raft_consensus.cc:2953] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
I20250904 19:59:20.977298 6579 raft_consensus.cc:2953] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
I20250904 19:59:20.979628 6359 raft_consensus.cc:2953] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
I20250904 19:59:20.996198 6072 catalog_manager.cc:5582] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc reported cstate change: config changed from index 11 to 12, 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d52eee06fc004fe2bb9d1d3619265627" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
I20250904 19:59:21.023020 2776 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 078bbd461f134ecb9712bedaf958058a
I20250904 19:59:21.023340 2776 log_verifier.cc:126] Checking tablet 8d6c85b5a6e04a1bb62574c5e27d4355
I20250904 19:59:21.117391 2776 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 8d6c85b5a6e04a1bb62574c5e27d4355
I20250904 19:59:21.117676 2776 log_verifier.cc:126] Checking tablet fffc60dadf624310ac4cb8f96f498653
I20250904 19:59:21.211810 2776 log_verifier.cc:177] Verified matching terms for 13 ops in tablet fffc60dadf624310ac4cb8f96f498653
I20250904 19:59:21.212364 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 6039
I20250904 19:59:21.238639 2776 minidump.cc:252] Setting minidump size limit to 20M
I20250904 19:59:21.240264 2776 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:21.241380 2776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:21.253228 6649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:21.253233 6650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:21.255777 6652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:21.362596 2776 server_base.cc:1047] running on GCE node
I20250904 19:59:21.364003 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250904 19:59:21.364240 2776 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250904 19:59:21.364382 2776 hybrid_clock.cc:648] HybridClock initialized: now 1757015961364368 us; error 0 us; skew 500 ppm
I20250904 19:59:21.365010 2776 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:21.368247 2776 webserver.cc:480] Webserver started at http://0.0.0.0:46487/ using document root <none> and password file <none>
I20250904 19:59:21.369097 2776 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:21.369292 2776 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:21.375365 2776 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250904 19:59:21.380017 6657 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:21.381158 2776 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250904 19:59:21.381508 2776 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "116b311348744b64b8e53eec1aad5435"
format_stamp: "Formatted at 2025-09-04 19:58:59 on dist-test-slave-tjsx"
I20250904 19:59:21.383409 2776 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:21.405774 2776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:21.408516 2776 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:21.409103 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:21.425269 2776 sys_catalog.cc:263] Verifying existing consensus state
W20250904 19:59:21.433056 2776 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.2.182.62:38495 exists but no master address supplied!
I20250904 19:59:21.436389 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap starting.
I20250904 19:59:21.500263 2776 log.cc:826] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:21.570834 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=13 ignored=0} mutations{seen=22 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:21.571610 2776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap complete.
I20250904 19:59:21.584939 2776 raft_consensus.cc:357] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:21.585620 2776 raft_consensus.cc:738] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116b311348744b64b8e53eec1aad5435, State: Initialized, Role: FOLLOWER
I20250904 19:59:21.586437 2776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:21.586989 2776 raft_consensus.cc:397] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:21.587270 2776 raft_consensus.cc:491] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:21.587615 2776 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 3 FOLLOWER]: Advancing to term 4
I20250904 19:59:21.593487 2776 raft_consensus.cc:513] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:21.594307 2776 leader_election.cc:304] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116b311348744b64b8e53eec1aad5435; no voters:
I20250904 19:59:21.595695 2776 leader_election.cc:290] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250904 19:59:21.596127 6664 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 4 FOLLOWER]: Leader election won for term 4
I20250904 19:59:21.597971 6664 raft_consensus.cc:695] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 4 LEADER]: Becoming Leader. State: Replica: 116b311348744b64b8e53eec1aad5435, State: Running, Role: LEADER
I20250904 19:59:21.599032 6664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:21.607800 6666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116b311348744b64b8e53eec1aad5435. Latest consensus state: current_term: 4 leader_uuid: "116b311348744b64b8e53eec1aad5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } } }
I20250904 19:59:21.608223 6665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "116b311348744b64b8e53eec1aad5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } } }
I20250904 19:59:21.608645 6666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: This master's current role is: LEADER
I20250904 19:59:21.608791 6665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: This master's current role is: LEADER
I20250904 19:59:21.636417 2776 tablet_replica.cc:331] stopping tablet replica
I20250904 19:59:21.637202 2776 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 4 LEADER]: Raft consensus shutting down.
I20250904 19:59:21.637804 2776 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250904 19:59:21.641144 2776 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250904 19:59:21.641880 2776 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250904 19:59:21.667213 2776 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250904 19:59:22.029086 6404 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38495: connect: Connection refused (error 111)
W20250904 19:59:22.031673 6572 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38495: connect: Connection refused (error 111)
W20250904 19:59:22.034111 6259 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38495 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38495: connect: Connection refused (error 111)
I20250904 19:59:26.767874 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 6109
I20250904 19:59:26.795673 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 6263
I20250904 19:59:26.822775 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 6408
I20250904 19:59:26.850236 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38495
--webserver_interface=127.2.182.62
--webserver_port=43493
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38495 with env {}
W20250904 19:59:27.162111 6739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:27.162806 6739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:27.163283 6739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:27.195114 6739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:59:27.195470 6739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:27.195724 6739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:59:27.195957 6739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:59:27.232216 6739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38495
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=43493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:27.233579 6739 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:27.235370 6739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:27.246554 6745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:27.248163 6746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:28.409307 6748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:28.411854 6747 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1159 milliseconds
I20250904 19:59:28.412040 6739 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:28.413353 6739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:28.416575 6739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:28.418082 6739 hybrid_clock.cc:648] HybridClock initialized: now 1757015968418033 us; error 65 us; skew 500 ppm
I20250904 19:59:28.419044 6739 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:28.425932 6739 webserver.cc:480] Webserver started at http://127.2.182.62:43493/ using document root <none> and password file <none>
I20250904 19:59:28.427038 6739 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:28.427277 6739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:28.435707 6739 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20250904 19:59:28.441191 6755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:28.442668 6739 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20250904 19:59:28.443081 6739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "116b311348744b64b8e53eec1aad5435"
format_stamp: "Formatted at 2025-09-04 19:58:59 on dist-test-slave-tjsx"
I20250904 19:59:28.445151 6739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:28.494028 6739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:28.495744 6739 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:28.496227 6739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:28.573755 6739 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38495
I20250904 19:59:28.573827 6806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:38495 every 8 connection(s)
I20250904 19:59:28.576784 6739 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:59:28.577813 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 6739
I20250904 19:59:28.580008 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:36843
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:38495
--webserver_port=35221
--webserver_interface=127.2.182.1
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:59:28.591344 6807 sys_catalog.cc:263] Verifying existing consensus state
I20250904 19:59:28.596984 6807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap starting.
I20250904 19:59:28.607827 6807 log.cc:826] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:28.693964 6807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=15 ignored=0} mutations{seen=24 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:28.694846 6807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Bootstrap complete.
I20250904 19:59:28.714107 6807 raft_consensus.cc:357] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:28.716444 6807 raft_consensus.cc:738] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116b311348744b64b8e53eec1aad5435, State: Initialized, Role: FOLLOWER
I20250904 19:59:28.717288 6807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:28.717821 6807 raft_consensus.cc:397] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:28.718083 6807 raft_consensus.cc:491] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:28.718361 6807 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 5 FOLLOWER]: Advancing to term 6
I20250904 19:59:28.723793 6807 raft_consensus.cc:513] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:28.724532 6807 leader_election.cc:304] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116b311348744b64b8e53eec1aad5435; no voters:
I20250904 19:59:28.726598 6807 leader_election.cc:290] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250904 19:59:28.727032 6811 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 6 FOLLOWER]: Leader election won for term 6
I20250904 19:59:28.730312 6811 raft_consensus.cc:695] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [term 6 LEADER]: Becoming Leader. State: Replica: 116b311348744b64b8e53eec1aad5435, State: Running, Role: LEADER
I20250904 19:59:28.731585 6807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:59:28.731246 6811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } }
I20250904 19:59:28.743564 6813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116b311348744b64b8e53eec1aad5435. Latest consensus state: current_term: 6 leader_uuid: "116b311348744b64b8e53eec1aad5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } } }
I20250904 19:59:28.744484 6813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: This master's current role is: LEADER
I20250904 19:59:28.743594 6812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "116b311348744b64b8e53eec1aad5435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116b311348744b64b8e53eec1aad5435" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38495 } } }
I20250904 19:59:28.745008 6812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435 [sys.catalog]: This master's current role is: LEADER
I20250904 19:59:28.759883 6818 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:59:28.784968 6818 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=71071a0dabcb4e2187ea57b1e2d74388]
I20250904 19:59:28.787113 6818 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=e097083716534c03aefc656d01943d7c]
I20250904 19:59:28.788889 6818 catalog_manager.cc:671] Loaded metadata for table TestTable [id=fb17c69f6d1b4733a52fe6f512bbf00a]
I20250904 19:59:28.798024 6818 tablet_loader.cc:96] loaded metadata for tablet 078bbd461f134ecb9712bedaf958058a (table TestTable2 [id=e097083716534c03aefc656d01943d7c])
I20250904 19:59:28.799677 6818 tablet_loader.cc:96] loaded metadata for tablet 8d6c85b5a6e04a1bb62574c5e27d4355 (table TestTable1 [id=71071a0dabcb4e2187ea57b1e2d74388])
I20250904 19:59:28.800935 6818 tablet_loader.cc:96] loaded metadata for tablet fffc60dadf624310ac4cb8f96f498653 (table TestTable [id=fb17c69f6d1b4733a52fe6f512bbf00a])
I20250904 19:59:28.802290 6818 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:59:28.808687 6818 catalog_manager.cc:1261] Loaded cluster ID: 0fe8823e3bc14e35badbe62ce3e59c2a
I20250904 19:59:28.809046 6818 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:59:28.810552 6828 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250904 19:59:28.818542 6818 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:59:28.825317 6818 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 116b311348744b64b8e53eec1aad5435: Loaded TSK: 0
I20250904 19:59:28.827414 6818 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250904 19:59:28.956840 6809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:28.957377 6809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:28.957919 6809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:28.992004 6809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:28.992954 6809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 19:59:29.030215 6809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:36843
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=35221
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:29.031687 6809 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:29.033573 6809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:29.047331 6834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:29.048322 6835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:30.206521 6837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:30.211266 6836 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1161 milliseconds
I20250904 19:59:30.211452 6809 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:30.212777 6809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:30.215298 6809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:30.216719 6809 hybrid_clock.cc:648] HybridClock initialized: now 1757015970216688 us; error 53 us; skew 500 ppm
I20250904 19:59:30.217656 6809 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:30.229190 6809 webserver.cc:480] Webserver started at http://127.2.182.1:35221/ using document root <none> and password file <none>
I20250904 19:59:30.230362 6809 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:30.230612 6809 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:30.240149 6809 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.005s
I20250904 19:59:30.247558 6844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:30.248981 6809 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250904 19:59:30.249397 6809 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "47edf36cc269417ba960d1c1955afde1"
format_stamp: "Formatted at 2025-09-04 19:59:01 on dist-test-slave-tjsx"
I20250904 19:59:30.251752 6809 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:30.310289 6809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:30.312006 6809 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:30.312525 6809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:30.315483 6809 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:30.323098 6851 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250904 19:59:30.342684 6809 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 19:59:30.343012 6809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.022s user 0.003s sys 0.000s
I20250904 19:59:30.343290 6809 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250904 19:59:30.349545 6851 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Bootstrap starting.
I20250904 19:59:30.352453 6809 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 19:59:30.352699 6809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.008s sys 0.000s
I20250904 19:59:30.423893 6851 log.cc:826] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:30.577792 6851 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:30.579026 6851 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Bootstrap complete.
I20250904 19:59:30.580971 6851 ts_tablet_manager.cc:1397] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Time spent bootstrapping tablet: real 0.232s user 0.174s sys 0.048s
I20250904 19:59:30.605669 6851 raft_consensus.cc:357] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:30.609541 6851 raft_consensus.cc:738] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47edf36cc269417ba960d1c1955afde1, State: Initialized, Role: FOLLOWER
I20250904 19:59:30.611027 6851 consensus_queue.cc:260] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:30.616705 6851 ts_tablet_manager.cc:1428] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1: Time spent starting tablet: real 0.035s user 0.031s sys 0.004s
I20250904 19:59:30.617693 6851 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Bootstrap starting.
I20250904 19:59:30.635336 6809 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36843
I20250904 19:59:30.635453 6959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:36843 every 8 connection(s)
I20250904 19:59:30.638736 6809 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 19:59:30.640424 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 6809
I20250904 19:59:30.642601 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:37583
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:38495
--webserver_port=42293
--webserver_interface=127.2.182.2
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:59:30.718822 6960 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:30.719415 6960 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:30.720799 6960 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:30.726991 6772 ts_manager.cc:194] Registered new tserver with Master: 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843)
I20250904 19:59:30.731662 6772 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) added, VOTER 47edf36cc269417ba960d1c1955afde1 (127.2.182.1) added, VOTER d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } } }
I20250904 19:59:30.786751 6851 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:30.787699 6851 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Bootstrap complete.
I20250904 19:59:30.788834 6772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:60639
I20250904 19:59:30.789335 6851 ts_tablet_manager.cc:1397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Time spent bootstrapping tablet: real 0.172s user 0.155s sys 0.012s
I20250904 19:59:30.791689 6851 raft_consensus.cc:357] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:30.792359 6851 raft_consensus.cc:738] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47edf36cc269417ba960d1c1955afde1, State: Initialized, Role: FOLLOWER
I20250904 19:59:30.793215 6851 consensus_queue.cc:260] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:30.794476 6960 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
I20250904 19:59:30.795395 6851 ts_tablet_manager.cc:1428] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20250904 19:59:31.039500 6961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:31.040073 6961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:31.040570 6961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:31.074307 6961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:31.075268 6961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 19:59:31.114188 6961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:37583
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=42293
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:31.115919 6961 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:31.117982 6961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:31.133443 6975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:31.884413 6981 raft_consensus.cc:491] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:59:31.885555 6981 raft_consensus.cc:513] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
W20250904 19:59:31.897946 6848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.2:37583: connect: Connection refused (error 111)
I20250904 19:59:31.904026 6981 leader_election.cc:290] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583), d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161)
W20250904 19:59:31.914103 6848 leader_election.cc:336] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583): Network error: Client connection negotiation failed: client connection to 127.2.182.2:37583: connect: Connection refused (error 111)
W20250904 19:59:31.915197 6846 leader_election.cc:336] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Network error: Client connection negotiation failed: client connection to 127.2.182.3:38161: connect: Connection refused (error 111)
I20250904 19:59:31.915913 6846 leader_election.cc:304] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47edf36cc269417ba960d1c1955afde1; no voters: 2e799fbd30c84f32af95370d7caf8fcc, d52eee06fc004fe2bb9d1d3619265627
I20250904 19:59:31.916909 6981 raft_consensus.cc:2747] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250904 19:59:32.655354 6984 raft_consensus.cc:491] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:59:32.656252 6984 raft_consensus.cc:513] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:32.668349 6984 leader_election.cc:290] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161), 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583)
W20250904 19:59:32.677264 6848 leader_election.cc:336] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583): Network error: Client connection negotiation failed: client connection to 127.2.182.2:37583: connect: Connection refused (error 111)
W20250904 19:59:32.678059 6846 leader_election.cc:336] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Network error: Client connection negotiation failed: client connection to 127.2.182.3:38161: connect: Connection refused (error 111)
I20250904 19:59:32.678532 6846 leader_election.cc:304] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47edf36cc269417ba960d1c1955afde1; no voters: 2e799fbd30c84f32af95370d7caf8fcc, d52eee06fc004fe2bb9d1d3619265627
I20250904 19:59:32.679812 6984 raft_consensus.cc:2747] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250904 19:59:32.534482 6974 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 6961
W20250904 19:59:32.707238 6974 kernel_stack_watchdog.cc:198] Thread 6961 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 397ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 19:59:31.134961 6976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:32.708335 6961 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.574s user 0.532s sys 1.019s
W20250904 19:59:32.708743 6961 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.574s user 0.532s sys 1.019s
W20250904 19:59:32.708909 6977 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1572 milliseconds
W20250904 19:59:32.710587 6978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:32.710541 6961 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:32.714594 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:32.717329 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:32.718901 6961 hybrid_clock.cc:648] HybridClock initialized: now 1757015972718836 us; error 58 us; skew 500 ppm
I20250904 19:59:32.719847 6961 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:32.726428 6961 webserver.cc:480] Webserver started at http://127.2.182.2:42293/ using document root <none> and password file <none>
I20250904 19:59:32.727504 6961 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:32.727744 6961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:32.736593 6961 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20250904 19:59:32.742542 6992 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:32.743917 6961 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250904 19:59:32.744310 6961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2e799fbd30c84f32af95370d7caf8fcc"
format_stamp: "Formatted at 2025-09-04 19:59:03 on dist-test-slave-tjsx"
I20250904 19:59:32.746426 6961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:32.797573 6961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:32.799154 6961 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:32.799628 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:32.802279 6961 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:32.808662 6999 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250904 19:59:32.826357 6961 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250904 19:59:32.826658 6961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.002s sys 0.000s
I20250904 19:59:32.827003 6961 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250904 19:59:32.833254 6999 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap starting.
I20250904 19:59:32.840158 6961 ts_tablet_manager.cc:610] Registered 3 tablets
I20250904 19:59:32.840449 6961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.013s user 0.012s sys 0.000s
I20250904 19:59:32.891986 6999 log.cc:826] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:33.017524 6961 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:37583
I20250904 19:59:33.017616 7106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:37583 every 8 connection(s)
I20250904 19:59:33.020314 6961 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250904 19:59:33.024335 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 6961
I20250904 19:59:33.025867 6999 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:33.026391 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:38161
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:38495
--webserver_port=40191
--webserver_interface=127.2.182.3
--builtin_ntp_servers=127.2.182.20:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250904 19:59:33.026990 6999 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap complete.
I20250904 19:59:33.028822 6999 ts_tablet_manager.cc:1397] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Time spent bootstrapping tablet: real 0.196s user 0.155s sys 0.037s
I20250904 19:59:33.050381 6999 raft_consensus.cc:357] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:33.054292 6999 raft_consensus.cc:738] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Initialized, Role: FOLLOWER
I20250904 19:59:33.055401 6999 consensus_queue.cc:260] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:33.059691 6999 ts_tablet_manager.cc:1428] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Time spent starting tablet: real 0.030s user 0.025s sys 0.004s
I20250904 19:59:33.060612 6999 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap starting.
I20250904 19:59:33.065455 7107 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:33.065940 7107 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:33.067044 7107 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:33.071226 6772 ts_manager.cc:194] Registered new tserver with Master: 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583)
I20250904 19:59:33.075026 6772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:37099
I20250904 19:59:33.078188 7107 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
I20250904 19:59:33.197000 6999 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:33.197726 6999 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap complete.
I20250904 19:59:33.198879 6999 ts_tablet_manager.cc:1397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Time spent bootstrapping tablet: real 0.138s user 0.122s sys 0.015s
I20250904 19:59:33.200536 6999 raft_consensus.cc:357] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:33.201002 6999 raft_consensus.cc:738] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Initialized, Role: FOLLOWER
I20250904 19:59:33.201488 6999 consensus_queue.cc:260] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:33.202895 6999 ts_tablet_manager.cc:1428] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250904 19:59:33.203589 6999 tablet_bootstrap.cc:492] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap starting.
I20250904 19:59:33.302068 6999 tablet_bootstrap.cc:492] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:33.302894 6999 tablet_bootstrap.cc:492] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Bootstrap complete.
I20250904 19:59:33.304030 6999 ts_tablet_manager.cc:1397] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Time spent bootstrapping tablet: real 0.101s user 0.083s sys 0.016s
I20250904 19:59:33.305708 6999 raft_consensus.cc:357] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:33.306092 6999 raft_consensus.cc:738] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Initialized, Role: FOLLOWER
I20250904 19:59:33.306582 6999 consensus_queue.cc:260] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:33.307039 6999 raft_consensus.cc:397] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:33.307312 6999 raft_consensus.cc:491] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:33.307621 6999 raft_consensus.cc:3058] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Advancing to term 3
I20250904 19:59:33.313227 6999 raft_consensus.cc:513] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:33.313984 6999 leader_election.cc:304] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e799fbd30c84f32af95370d7caf8fcc; no voters:
I20250904 19:59:33.314564 6999 leader_election.cc:290] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 3 election: Requested vote from peers
I20250904 19:59:33.314841 7112 raft_consensus.cc:2802] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Leader election won for term 3
I20250904 19:59:33.318050 6999 ts_tablet_manager.cc:1428] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20250904 19:59:33.318683 7112 raft_consensus.cc:695] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [term 3 LEADER]: Becoming Leader. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Running, Role: LEADER
I20250904 19:59:33.319463 7112 consensus_queue.cc:237] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } }
I20250904 19:59:33.332149 6772 catalog_manager.cc:5582] T 078bbd461f134ecb9712bedaf958058a P 2e799fbd30c84f32af95370d7caf8fcc reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "2e799fbd30c84f32af95370d7caf8fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } health_report { overall_health: HEALTHY } } }
W20250904 19:59:33.412705 7111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:33.413206 7111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:33.413733 7111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:33.445902 7111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:33.446779 7111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 19:59:33.485373 7111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:38161
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=40191
--tserver_master_addrs=127.2.182.62:38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:33.486668 7111 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:33.488377 7111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:33.501093 7128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:33.802253 7134 raft_consensus.cc:491] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:59:33.802989 7134 raft_consensus.cc:513] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:33.846268 7134 leader_election.cc:290] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583), d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161)
W20250904 19:59:33.861815 6846 leader_election.cc:336] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Network error: Client connection negotiation failed: client connection to 127.2.182.3:38161: connect: Connection refused (error 111)
I20250904 19:59:33.876865 7062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffc60dadf624310ac4cb8f96f498653" candidate_uuid: "47edf36cc269417ba960d1c1955afde1" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "2e799fbd30c84f32af95370d7caf8fcc" is_pre_election: true
I20250904 19:59:33.877838 7062 raft_consensus.cc:2466] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47edf36cc269417ba960d1c1955afde1 in term 2.
I20250904 19:59:33.880101 6848 leader_election.cc:304] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e799fbd30c84f32af95370d7caf8fcc, 47edf36cc269417ba960d1c1955afde1; no voters: d52eee06fc004fe2bb9d1d3619265627
I20250904 19:59:33.881515 7134 raft_consensus.cc:2802] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 19:59:33.881892 7134 raft_consensus.cc:491] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 19:59:33.882196 7134 raft_consensus.cc:3058] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Advancing to term 3
I20250904 19:59:33.899830 7134 raft_consensus.cc:513] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:33.906268 7062 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fffc60dadf624310ac4cb8f96f498653" candidate_uuid: "47edf36cc269417ba960d1c1955afde1" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "2e799fbd30c84f32af95370d7caf8fcc"
I20250904 19:59:33.906949 7062 raft_consensus.cc:3058] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Advancing to term 3
I20250904 19:59:33.914484 7062 raft_consensus.cc:2466] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47edf36cc269417ba960d1c1955afde1 in term 3.
I20250904 19:59:33.916078 6848 leader_election.cc:304] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e799fbd30c84f32af95370d7caf8fcc, 47edf36cc269417ba960d1c1955afde1; no voters:
W20250904 19:59:33.927117 6846 leader_election.cc:336] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Network error: Client connection negotiation failed: client connection to 127.2.182.3:38161: connect: Connection refused (error 111)
I20250904 19:59:33.928647 7134 leader_election.cc:290] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [CANDIDATE]: Term 3 election: Requested vote from peers 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583), d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161)
I20250904 19:59:33.929706 7134 raft_consensus.cc:2802] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 3 FOLLOWER]: Leader election won for term 3
I20250904 19:59:33.930989 7134 raft_consensus.cc:695] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 3 LEADER]: Becoming Leader. State: Replica: 47edf36cc269417ba960d1c1955afde1, State: Running, Role: LEADER
I20250904 19:59:33.932603 7134 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:33.960520 6772 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 reported cstate change: term changed from 2 to 3, leader changed from <none> to 47edf36cc269417ba960d1c1955afde1 (127.2.182.1). New cstate: current_term: 3 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250904 19:59:34.370987 6846 consensus_peers.cc:489] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 -> Peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Couldn't send request to peer d52eee06fc004fe2bb9d1d3619265627. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.3:38161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 19:59:34.389165 7062 raft_consensus.cc:1273] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Refusing update from remote peer 47edf36cc269417ba960d1c1955afde1: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250904 19:59:34.391736 7134 consensus_queue.cc:1035] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250904 19:59:34.516990 6914 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:34.526343 7061 raft_consensus.cc:1273] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Refusing update from remote peer 47edf36cc269417ba960d1c1955afde1: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250904 19:59:34.528312 7134 consensus_queue.cc:1035] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20250904 19:59:34.537222 7140 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
I20250904 19:59:34.545231 7062 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
I20250904 19:59:34.566816 6758 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fffc60dadf624310ac4cb8f96f498653 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250904 19:59:34.575544 6772 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc reported cstate change: config changed from index 13 to 15, VOTER d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) evicted. New cstate: current_term: 3 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } }
W20250904 19:59:34.608021 6772 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet fffc60dadf624310ac4cb8f96f498653 on TS d52eee06fc004fe2bb9d1d3619265627: Not found: failed to reset TS proxy: Could not find TS for UUID d52eee06fc004fe2bb9d1d3619265627
I20250904 19:59:34.616431 6914 consensus_queue.cc:237] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } }
I20250904 19:59:34.622251 7140 raft_consensus.cc:2953] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } }
I20250904 19:59:34.633371 6758 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fffc60dadf624310ac4cb8f96f498653 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250904 19:59:34.639585 6771 catalog_manager.cc:5582] T fffc60dadf624310ac4cb8f96f498653 P 47edf36cc269417ba960d1c1955afde1 reported cstate change: config changed from index 15 to 16, VOTER 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2) evicted. New cstate: current_term: 3 leader_uuid: "47edf36cc269417ba960d1c1955afde1" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } health_report { overall_health: HEALTHY } } }
W20250904 19:59:34.641911 7103 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5060 after lost signal to thread 6970
W20250904 19:59:34.661931 6757 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet fffc60dadf624310ac4cb8f96f498653 on TS d52eee06fc004fe2bb9d1d3619265627 failed: Not found: failed to reset TS proxy: Could not find TS for UUID d52eee06fc004fe2bb9d1d3619265627
I20250904 19:59:34.709877 7042 tablet_service.cc:1515] Processing DeleteTablet for tablet fffc60dadf624310ac4cb8f96f498653 with delete_type TABLET_DATA_TOMBSTONED (TS 2e799fbd30c84f32af95370d7caf8fcc not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:57680
I20250904 19:59:34.730422 7155 tablet_replica.cc:331] stopping tablet replica
I20250904 19:59:34.731480 7155 raft_consensus.cc:2241] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Raft consensus shutting down.
I20250904 19:59:34.732189 7155 raft_consensus.cc:2270] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Raft consensus is shut down!
I20250904 19:59:34.736246 7155 ts_tablet_manager.cc:1905] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250904 19:59:34.736932 7145 raft_consensus.cc:491] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 19:59:34.737592 7145 raft_consensus.cc:513] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:34.745393 7145 leader_election.cc:290] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161), 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843)
I20250904 19:59:34.759886 7155 ts_tablet_manager.cc:1918] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250904 19:59:34.760471 7155 log.cc:1199] T fffc60dadf624310ac4cb8f96f498653 P 2e799fbd30c84f32af95370d7caf8fcc: Deleting WAL directory at /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/wals/fffc60dadf624310ac4cb8f96f498653
I20250904 19:59:34.764300 6759 catalog_manager.cc:4928] TS 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2:37583): tablet fffc60dadf624310ac4cb8f96f498653 (table TestTable [id=fb17c69f6d1b4733a52fe6f512bbf00a]) successfully deleted
W20250904 19:59:34.765489 6994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.3:38161: connect: Connection refused (error 111)
W20250904 19:59:34.777503 6994 leader_election.cc:336] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Network error: Client connection negotiation failed: client connection to 127.2.182.3:38161: connect: Connection refused (error 111)
I20250904 19:59:34.787192 6914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d6c85b5a6e04a1bb62574c5e27d4355" candidate_uuid: "2e799fbd30c84f32af95370d7caf8fcc" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "47edf36cc269417ba960d1c1955afde1" is_pre_election: true
I20250904 19:59:34.788262 6914 raft_consensus.cc:2466] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e799fbd30c84f32af95370d7caf8fcc in term 2.
I20250904 19:59:34.790457 6995 leader_election.cc:304] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e799fbd30c84f32af95370d7caf8fcc, 47edf36cc269417ba960d1c1955afde1; no voters: d52eee06fc004fe2bb9d1d3619265627
I20250904 19:59:34.792244 7145 raft_consensus.cc:2802] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250904 19:59:34.792627 7145 raft_consensus.cc:491] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 19:59:34.792980 7145 raft_consensus.cc:3058] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 2 FOLLOWER]: Advancing to term 3
I20250904 19:59:34.802384 7145 raft_consensus.cc:513] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:34.807122 6914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d6c85b5a6e04a1bb62574c5e27d4355" candidate_uuid: "2e799fbd30c84f32af95370d7caf8fcc" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "47edf36cc269417ba960d1c1955afde1"
I20250904 19:59:34.807921 6914 raft_consensus.cc:3058] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 2 FOLLOWER]: Advancing to term 3
W20250904 19:59:34.809604 6994 leader_election.cc:336] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Network error: Client connection negotiation failed: client connection to 127.2.182.3:38161: connect: Connection refused (error 111)
I20250904 19:59:34.816779 6914 raft_consensus.cc:2466] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e799fbd30c84f32af95370d7caf8fcc in term 3.
I20250904 19:59:34.817482 7145 leader_election.cc:290] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 3 election: Requested vote from peers d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161), 47edf36cc269417ba960d1c1955afde1 (127.2.182.1:36843)
I20250904 19:59:34.820035 6995 leader_election.cc:304] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e799fbd30c84f32af95370d7caf8fcc, 47edf36cc269417ba960d1c1955afde1; no voters: d52eee06fc004fe2bb9d1d3619265627
I20250904 19:59:34.830936 7145 raft_consensus.cc:2802] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 FOLLOWER]: Leader election won for term 3
I20250904 19:59:34.834164 7145 raft_consensus.cc:695] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [term 3 LEADER]: Becoming Leader. State: Replica: 2e799fbd30c84f32af95370d7caf8fcc, State: Running, Role: LEADER
I20250904 19:59:34.837297 7145 consensus_queue.cc:237] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:34.856712 6771 catalog_manager.cc:5582] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc reported cstate change: term changed from 2 to 3, leader changed from d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3) to 2e799fbd30c84f32af95370d7caf8fcc (127.2.182.2). New cstate: current_term: 3 leader_uuid: "2e799fbd30c84f32af95370d7caf8fcc" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250904 19:59:34.903625 7127 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 7111
W20250904 19:59:35.022303 7127 kernel_stack_watchdog.cc:198] Thread 7111 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 19:59:33.501679 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:35.023888 7111 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.523s user 0.552s sys 0.745s
W20250904 19:59:35.024443 7111 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.523s user 0.553s sys 0.745s
I20250904 19:59:35.024791 7111 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250904 19:59:35.025434 7130 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1521 milliseconds
W20250904 19:59:35.024873 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:35.029634 7111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:35.032083 7111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:35.033581 7111 hybrid_clock.cc:648] HybridClock initialized: now 1757015975033512 us; error 61 us; skew 500 ppm
I20250904 19:59:35.034454 7111 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:35.041297 7111 webserver.cc:480] Webserver started at http://127.2.182.3:40191/ using document root <none> and password file <none>
I20250904 19:59:35.042343 7111 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:35.042603 7111 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:35.051453 7111 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250904 19:59:35.057070 7165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:35.058650 7111 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250904 19:59:35.059073 7111 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d52eee06fc004fe2bb9d1d3619265627"
format_stamp: "Formatted at 2025-09-04 19:59:04 on dist-test-slave-tjsx"
I20250904 19:59:35.061376 7111 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:35.119764 7111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:35.121513 7111 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:35.122031 7111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:35.125174 7111 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 19:59:35.132030 7172 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250904 19:59:35.151186 7111 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250904 19:59:35.151479 7111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.000s sys 0.002s
I20250904 19:59:35.151752 7111 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250904 19:59:35.157655 7172 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap starting.
I20250904 19:59:35.162752 7111 ts_tablet_manager.cc:610] Registered 2 tablets
I20250904 19:59:35.163064 7111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.007s sys 0.005s
I20250904 19:59:35.233137 7172 log.cc:826] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:35.382515 6914 raft_consensus.cc:1273] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 47edf36cc269417ba960d1c1955afde1 [term 3 FOLLOWER]: Refusing update from remote peer 2e799fbd30c84f32af95370d7caf8fcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250904 19:59:35.384788 7275 consensus_queue.cc:1035] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250904 19:59:35.388178 7172 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:35.389477 7172 tablet_bootstrap.cc:492] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap complete.
I20250904 19:59:35.391497 7172 ts_tablet_manager.cc:1397] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Time spent bootstrapping tablet: real 0.234s user 0.184s sys 0.042s
I20250904 19:59:35.418591 7172 raft_consensus.cc:357] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:35.422390 7172 raft_consensus.cc:738] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52eee06fc004fe2bb9d1d3619265627, State: Initialized, Role: FOLLOWER
I20250904 19:59:35.423561 7172 consensus_queue.cc:260] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } } peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } attrs { promote: false } }
I20250904 19:59:35.437319 7172 ts_tablet_manager.cc:1428] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Time spent starting tablet: real 0.045s user 0.030s sys 0.008s
I20250904 19:59:35.438423 7172 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap starting.
I20250904 19:59:35.462414 7289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:38161 every 8 connection(s)
I20250904 19:59:35.462971 7111 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:38161
I20250904 19:59:35.467933 7111 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250904 19:59:35.475360 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 7111
W20250904 19:59:35.530742 6994 consensus_peers.cc:489] T 8d6c85b5a6e04a1bb62574c5e27d4355 P 2e799fbd30c84f32af95370d7caf8fcc -> Peer d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): Couldn't send request to peer d52eee06fc004fe2bb9d1d3619265627. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250904 19:59:35.535558 7292 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38495
I20250904 19:59:35.536123 7292 heartbeater.cc:461] Registering TS with master...
I20250904 19:59:35.537611 7292 heartbeater.cc:507] Master 127.2.182.62:38495 requested a full tablet report, sending...
I20250904 19:59:35.542670 6771 ts_manager.cc:194] Registered new tserver with Master: d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161)
I20250904 19:59:35.547397 6771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:52359
I20250904 19:59:35.557298 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 19:59:35.563174 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250904 19:59:35.567641 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250904 19:59:35.625212 7172 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 19:59:35.626274 7172 tablet_bootstrap.cc:492] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Bootstrap complete.
I20250904 19:59:35.628055 7172 ts_tablet_manager.cc:1397] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Time spent bootstrapping tablet: real 0.190s user 0.127s sys 0.033s
I20250904 19:59:35.630378 7172 raft_consensus.cc:357] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:35.631148 7172 raft_consensus.cc:738] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52eee06fc004fe2bb9d1d3619265627, State: Initialized, Role: FOLLOWER
I20250904 19:59:35.631892 7172 consensus_queue.cc:260] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d52eee06fc004fe2bb9d1d3619265627" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 38161 } } peers { permanent_uuid: "47edf36cc269417ba960d1c1955afde1" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 36843 } attrs { promote: false } } peers { permanent_uuid: "2e799fbd30c84f32af95370d7caf8fcc" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 37583 } attrs { promote: false } }
I20250904 19:59:35.635317 7292 heartbeater.cc:499] Master 127.2.182.62:38495 was elected leader, sending a full tablet report...
I20250904 19:59:35.636202 7172 ts_tablet_manager.cc:1428] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250904 19:59:35.841367 7215 tablet_service.cc:1515] Processing DeleteTablet for tablet fffc60dadf624310ac4cb8f96f498653 with delete_type TABLET_DATA_TOMBSTONED (TS d52eee06fc004fe2bb9d1d3619265627 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:50186
I20250904 19:59:35.844183 7301 tablet_replica.cc:331] stopping tablet replica
I20250904 19:59:35.844936 7301 raft_consensus.cc:2241] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250904 19:59:35.845472 7301 raft_consensus.cc:2270] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250904 19:59:35.847980 7301 ts_tablet_manager.cc:1905] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250904 19:59:35.861307 7301 ts_tablet_manager.cc:1918] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250904 19:59:35.861688 7301 log.cc:1199] T fffc60dadf624310ac4cb8f96f498653 P d52eee06fc004fe2bb9d1d3619265627: Deleting WAL directory at /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/wals/fffc60dadf624310ac4cb8f96f498653
I20250904 19:59:35.863462 6757 catalog_manager.cc:4928] TS d52eee06fc004fe2bb9d1d3619265627 (127.2.182.3:38161): tablet fffc60dadf624310ac4cb8f96f498653 (table TestTable [id=fb17c69f6d1b4733a52fe6f512bbf00a]) successfully deleted
I20250904 19:59:36.017120 7235 raft_consensus.cc:3058] T 8d6c85b5a6e04a1bb62574c5e27d4355 P d52eee06fc004fe2bb9d1d3619265627 [term 2 FOLLOWER]: Advancing to term 3
W20250904 19:59:36.573231 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:37.579165 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:38.583834 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:39.588279 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:40.592578 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:41.596323 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:42.600706 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:43.604791 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:44.609059 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:45.612939 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:46.617674 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:47.622134 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:48.626168 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:49.630007 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:50.633610 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:51.637539 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:52.641607 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:53.645974 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250904 19:59:54.649742 2776 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet fffc60dadf624310ac4cb8f96f498653: tablet_id: "fffc60dadf624310ac4cb8f96f498653" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250904 19:59:55.653213 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 6809
I20250904 19:59:55.680083 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 6961
I20250904 19:59:55.708112 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 7111
I20250904 19:59:55.731863 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 6739
2025-09-04T19:59:55Z chronyd exiting
I20250904 19:59:55.781612 2776 test_util.cc:183] -----------------------------------------------
I20250904 19:59:55.781903 2776 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757015875386078-2776-0
[ FAILED ] AdminCliTest.TestRebuildTables (58278 ms)
[----------] 5 tests from AdminCliTest (120330 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250904 19:59:55.786217 2776 test_util.cc:276] Using random seed: 709567437
I20250904 19:59:55.790892 2776 ts_itest-base.cc:115] Starting cluster with:
I20250904 19:59:55.791131 2776 ts_itest-base.cc:116] --------------
I20250904 19:59:55.791309 2776 ts_itest-base.cc:117] 5 tablet servers
I20250904 19:59:55.791469 2776 ts_itest-base.cc:118] 3 replicas per TS
I20250904 19:59:55.791621 2776 ts_itest-base.cc:119] --------------
2025-09-04T19:59:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T19:59:55Z Disabled control of system clock
I20250904 19:59:55.840627 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38291
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38291
--raft_prepare_replacement_before_eviction=true with env {}
W20250904 19:59:56.158638 7328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:56.159242 7328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:56.159675 7328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:56.191473 7328 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250904 19:59:56.191848 7328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 19:59:56.192059 7328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:56.192258 7328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 19:59:56.192453 7328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 19:59:56.228736 7328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38291
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:56.230077 7328 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:56.231835 7328 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:56.244063 7334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:56.244756 7335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:57.525925 7337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:57.528234 7336 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1278 milliseconds
W20250904 19:59:57.528861 7328 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.285s user 0.543s sys 0.741s
W20250904 19:59:57.529124 7328 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.286s user 0.543s sys 0.741s
I20250904 19:59:57.529309 7328 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:57.530407 7328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:57.533353 7328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:57.534987 7328 hybrid_clock.cc:648] HybridClock initialized: now 1757015997534885 us; error 78 us; skew 500 ppm
I20250904 19:59:57.536075 7328 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:57.545408 7328 webserver.cc:480] Webserver started at http://127.2.182.62:33857/ using document root <none> and password file <none>
I20250904 19:59:57.546579 7328 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:57.547124 7328 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:57.547719 7328 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:59:57.552703 7328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4511996fd98849069f3e0db2e42635d7"
format_stamp: "Formatted at 2025-09-04 19:59:57 on dist-test-slave-tjsx"
I20250904 19:59:57.553979 7328 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4511996fd98849069f3e0db2e42635d7"
format_stamp: "Formatted at 2025-09-04 19:59:57 on dist-test-slave-tjsx"
I20250904 19:59:57.563735 7328 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.002s
I20250904 19:59:57.570554 7344 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:57.572279 7328 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250904 19:59:57.572773 7328 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4511996fd98849069f3e0db2e42635d7"
format_stamp: "Formatted at 2025-09-04 19:59:57 on dist-test-slave-tjsx"
I20250904 19:59:57.573174 7328 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 19:59:57.646525 7328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 19:59:57.648245 7328 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 19:59:57.648751 7328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 19:59:57.722064 7328 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38291
I20250904 19:59:57.722095 7395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:38291 every 8 connection(s)
I20250904 19:59:57.725014 7328 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 19:59:57.730587 7396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 19:59:57.732729 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 7328
I20250904 19:59:57.733157 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250904 19:59:57.755013 7396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7: Bootstrap starting.
I20250904 19:59:57.761466 7396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7: Neither blocks nor log segments found. Creating new log.
I20250904 19:59:57.763808 7396 log.cc:826] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7: Log is configured to *not* fsync() on all Append() calls
I20250904 19:59:57.768775 7396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7: No bootstrap required, opened a new log
I20250904 19:59:57.787873 7396 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4511996fd98849069f3e0db2e42635d7" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38291 } }
I20250904 19:59:57.788609 7396 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 19:59:57.788846 7396 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4511996fd98849069f3e0db2e42635d7, State: Initialized, Role: FOLLOWER
I20250904 19:59:57.789673 7396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4511996fd98849069f3e0db2e42635d7" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38291 } }
I20250904 19:59:57.790338 7396 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 19:59:57.790623 7396 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 19:59:57.790983 7396 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 19:59:57.795186 7396 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4511996fd98849069f3e0db2e42635d7" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38291 } }
I20250904 19:59:57.796036 7396 leader_election.cc:304] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4511996fd98849069f3e0db2e42635d7; no voters:
I20250904 19:59:57.797758 7396 leader_election.cc:290] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 19:59:57.798596 7401 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 19:59:57.800853 7401 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [term 1 LEADER]: Becoming Leader. State: Replica: 4511996fd98849069f3e0db2e42635d7, State: Running, Role: LEADER
I20250904 19:59:57.801949 7401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4511996fd98849069f3e0db2e42635d7" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38291 } }
I20250904 19:59:57.803798 7396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 19:59:57.814414 7402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4511996fd98849069f3e0db2e42635d7. Latest consensus state: current_term: 1 leader_uuid: "4511996fd98849069f3e0db2e42635d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4511996fd98849069f3e0db2e42635d7" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38291 } } }
I20250904 19:59:57.815694 7402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [sys.catalog]: This master's current role is: LEADER
I20250904 19:59:57.815299 7403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4511996fd98849069f3e0db2e42635d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4511996fd98849069f3e0db2e42635d7" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38291 } } }
I20250904 19:59:57.816790 7403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7 [sys.catalog]: This master's current role is: LEADER
I20250904 19:59:57.820188 7409 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 19:59:57.832289 7409 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 19:59:57.851989 7409 catalog_manager.cc:1349] Generated new cluster ID: e5804bedf65645b5a0bc4f89b10eb836
I20250904 19:59:57.852348 7409 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 19:59:57.877130 7409 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 19:59:57.879323 7409 catalog_manager.cc:1506] Loading token signing keys...
I20250904 19:59:57.897150 7409 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4511996fd98849069f3e0db2e42635d7: Generated new TSK 0
I20250904 19:59:57.898365 7409 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 19:59:57.908264 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--builtin_ntp_servers=127.2.182.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250904 19:59:58.245713 7420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 19:59:58.246335 7420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 19:59:58.246924 7420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 19:59:58.282146 7420 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250904 19:59:58.282650 7420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 19:59:58.283569 7420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 19:59:58.323329 7420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 19:59:58.325032 7420 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 19:59:58.327072 7420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 19:59:58.341990 7426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:59.765442 7392 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 7329
W20250904 19:59:59.766340 7392 debug-util.cc:398] Leaking SignalData structure 0x7b080006f7a0 after lost signal to thread 7395
W20250904 19:59:58.343916 7427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 19:59:59.744791 7425 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7420
W20250904 19:59:59.900285 7420 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.558s user 0.688s sys 0.748s
W20250904 19:59:59.900678 7420 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.559s user 0.688s sys 0.748s
W20250904 19:59:59.905440 7428 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1562 milliseconds
W20250904 19:59:59.906042 7429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 19:59:59.906008 7420 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 19:59:59.907536 7420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 19:59:59.911957 7420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 19:59:59.913479 7420 hybrid_clock.cc:648] HybridClock initialized: now 1757015999913396 us; error 97 us; skew 500 ppm
I20250904 19:59:59.914358 7420 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 19:59:59.921571 7420 webserver.cc:480] Webserver started at http://127.2.182.1:38401/ using document root <none> and password file <none>
I20250904 19:59:59.923385 7420 fs_manager.cc:362] Metadata directory not provided
I20250904 19:59:59.923730 7420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 19:59:59.924260 7420 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 19:59:59.929369 7420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "19cba5e4ee054fe8958aaf8eda5b434b"
format_stamp: "Formatted at 2025-09-04 19:59:59 on dist-test-slave-tjsx"
I20250904 19:59:59.930672 7420 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "19cba5e4ee054fe8958aaf8eda5b434b"
format_stamp: "Formatted at 2025-09-04 19:59:59 on dist-test-slave-tjsx"
I20250904 19:59:59.938995 7420 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20250904 19:59:59.945502 7436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 19:59:59.946771 7420 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.001s
I20250904 19:59:59.947141 7420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "19cba5e4ee054fe8958aaf8eda5b434b"
format_stamp: "Formatted at 2025-09-04 19:59:59 on dist-test-slave-tjsx"
I20250904 19:59:59.947506 7420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:00.000847 7420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:00.002453 7420 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:00.002972 7420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:00.006270 7420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:00.011202 7420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:00.011456 7420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:00.011737 7420 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:00.011898 7420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:00.200115 7420 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:36553
I20250904 20:00:00.200212 7548 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:36553 every 8 connection(s)
I20250904 20:00:00.203013 7420 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 20:00:00.211738 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 7420
I20250904 20:00:00.212301 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250904 20:00:00.225190 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--builtin_ntp_servers=127.2.182.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250904 20:00:00.274358 7549 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38291
I20250904 20:00:00.274878 7549 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:00.276032 7549 heartbeater.cc:507] Master 127.2.182.62:38291 requested a full tablet report, sending...
I20250904 20:00:00.279304 7361 ts_manager.cc:194] Registered new tserver with Master: 19cba5e4ee054fe8958aaf8eda5b434b (127.2.182.1:36553)
I20250904 20:00:00.281550 7361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:33869
W20250904 20:00:00.607219 7553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:00.608029 7553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:00.608848 7553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:00.647419 7553 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250904 20:00:00.647851 7553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:00.648670 7553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 20:00:00.686288 7553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:00.687772 7553 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:00.689571 7553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:00.703073 7559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:01.285658 7549 heartbeater.cc:499] Master 127.2.182.62:38291 was elected leader, sending a full tablet report...
W20250904 20:00:00.703958 7560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:02.091212 7553 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.389s user 0.552s sys 0.836s
W20250904 20:00:02.091599 7553 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.390s user 0.552s sys 0.836s
I20250904 20:00:02.094734 7553 server_base.cc:1047] running on GCE node
W20250904 20:00:02.095260 7564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:02.096763 7553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:02.099709 7553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:02.101282 7553 hybrid_clock.cc:648] HybridClock initialized: now 1757016002101201 us; error 89 us; skew 500 ppm
I20250904 20:00:02.102455 7553 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:02.111889 7553 webserver.cc:480] Webserver started at http://127.2.182.2:39209/ using document root <none> and password file <none>
I20250904 20:00:02.113477 7553 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:02.113811 7553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:02.114459 7553 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:02.121886 7553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "12d16bacd0f64f5aaa585f5a63083dee"
format_stamp: "Formatted at 2025-09-04 20:00:02 on dist-test-slave-tjsx"
I20250904 20:00:02.123695 7553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "12d16bacd0f64f5aaa585f5a63083dee"
format_stamp: "Formatted at 2025-09-04 20:00:02 on dist-test-slave-tjsx"
I20250904 20:00:02.134651 7553 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.004s sys 0.006s
I20250904 20:00:02.143435 7569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:02.144997 7553 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20250904 20:00:02.145570 7553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "12d16bacd0f64f5aaa585f5a63083dee"
format_stamp: "Formatted at 2025-09-04 20:00:02 on dist-test-slave-tjsx"
I20250904 20:00:02.146144 7553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:02.223307 7553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:02.225391 7553 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:02.225999 7553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:02.229501 7553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:02.235634 7553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:02.235956 7553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:02.236266 7553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:02.236488 7553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:02.401384 7553 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:45101
I20250904 20:00:02.401451 7681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:45101 every 8 connection(s)
I20250904 20:00:02.405382 7553 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250904 20:00:02.411104 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 7553
I20250904 20:00:02.411657 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250904 20:00:02.422044 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--builtin_ntp_servers=127.2.182.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250904 20:00:02.454721 7682 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38291
I20250904 20:00:02.455305 7682 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:02.456787 7682 heartbeater.cc:507] Master 127.2.182.62:38291 requested a full tablet report, sending...
I20250904 20:00:02.459931 7361 ts_manager.cc:194] Registered new tserver with Master: 12d16bacd0f64f5aaa585f5a63083dee (127.2.182.2:45101)
I20250904 20:00:02.461788 7361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:50933
W20250904 20:00:02.751258 7686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:02.751880 7686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:02.752478 7686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:02.787037 7686 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250904 20:00:02.787604 7686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:02.788457 7686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 20:00:02.826889 7686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:02.828281 7686 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:02.830051 7686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:02.842772 7692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:03.466379 7682 heartbeater.cc:499] Master 127.2.182.62:38291 was elected leader, sending a full tablet report...
W20250904 20:00:02.844596 7693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:03.955515 7695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:03.957576 7694 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1110 milliseconds
I20250904 20:00:03.957687 7686 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:03.958986 7686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:03.961639 7686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:03.963091 7686 hybrid_clock.cc:648] HybridClock initialized: now 1757016003963048 us; error 60 us; skew 500 ppm
I20250904 20:00:03.963918 7686 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:03.972075 7686 webserver.cc:480] Webserver started at http://127.2.182.3:37995/ using document root <none> and password file <none>
I20250904 20:00:03.973389 7686 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:03.973706 7686 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:03.974272 7686 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:03.978958 7686 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "eee7037a4c6247db83f6a139b9b8610a"
format_stamp: "Formatted at 2025-09-04 20:00:03 on dist-test-slave-tjsx"
I20250904 20:00:03.980155 7686 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "eee7037a4c6247db83f6a139b9b8610a"
format_stamp: "Formatted at 2025-09-04 20:00:03 on dist-test-slave-tjsx"
I20250904 20:00:03.988377 7686 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250904 20:00:03.995474 7702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:03.996915 7686 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250904 20:00:03.997316 7686 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "eee7037a4c6247db83f6a139b9b8610a"
format_stamp: "Formatted at 2025-09-04 20:00:03 on dist-test-slave-tjsx"
I20250904 20:00:03.997731 7686 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:04.061244 7686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:04.063117 7686 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:04.063638 7686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:04.066591 7686 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:04.071621 7686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:04.071883 7686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:04.072157 7686 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:04.072324 7686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:04.226029 7686 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:42757
I20250904 20:00:04.226356 7814 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:42757 every 8 connection(s)
I20250904 20:00:04.229132 7686 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250904 20:00:04.232245 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 7686
I20250904 20:00:04.232815 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250904 20:00:04.240520 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.4:0
--local_ip_for_outbound_sockets=127.2.182.4
--webserver_interface=127.2.182.4
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--builtin_ntp_servers=127.2.182.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250904 20:00:04.261633 7815 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38291
I20250904 20:00:04.262180 7815 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:04.263619 7815 heartbeater.cc:507] Master 127.2.182.62:38291 requested a full tablet report, sending...
I20250904 20:00:04.266633 7361 ts_manager.cc:194] Registered new tserver with Master: eee7037a4c6247db83f6a139b9b8610a (127.2.182.3:42757)
I20250904 20:00:04.268106 7361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:60077
W20250904 20:00:04.577752 7819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:04.578382 7819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:04.578991 7819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:04.614837 7819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250904 20:00:04.615298 7819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:04.616138 7819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.4
I20250904 20:00:04.654407 7819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.2.182.4
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.4
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:04.656077 7819 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:04.657975 7819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:04.672797 7825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:05.272073 7815 heartbeater.cc:499] Master 127.2.182.62:38291 was elected leader, sending a full tablet report...
W20250904 20:00:04.674738 7826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:05.764747 7827 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250904 20:00:05.766106 7828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:05.766137 7819 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:05.769729 7819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:05.772058 7819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:05.773491 7819 hybrid_clock.cc:648] HybridClock initialized: now 1757016005773424 us; error 74 us; skew 500 ppm
I20250904 20:00:05.774382 7819 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:05.781042 7819 webserver.cc:480] Webserver started at http://127.2.182.4:33887/ using document root <none> and password file <none>
I20250904 20:00:05.782065 7819 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:05.782299 7819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:05.782862 7819 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:05.787664 7819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5ba1e6971c6c42a19e7b5c96ef0665e5"
format_stamp: "Formatted at 2025-09-04 20:00:05 on dist-test-slave-tjsx"
I20250904 20:00:05.788997 7819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5ba1e6971c6c42a19e7b5c96ef0665e5"
format_stamp: "Formatted at 2025-09-04 20:00:05 on dist-test-slave-tjsx"
I20250904 20:00:05.796829 7819 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250904 20:00:05.803100 7836 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:05.804316 7819 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250904 20:00:05.804689 7819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5ba1e6971c6c42a19e7b5c96ef0665e5"
format_stamp: "Formatted at 2025-09-04 20:00:05 on dist-test-slave-tjsx"
I20250904 20:00:05.805087 7819 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:05.858613 7819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:05.860195 7819 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:05.860661 7819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:05.863294 7819 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:05.867695 7819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:05.867954 7819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:05.868216 7819 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:05.868372 7819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:06.007417 7819 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.4:45799
I20250904 20:00:06.007615 7948 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.4:45799 every 8 connection(s)
I20250904 20:00:06.010288 7819 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250904 20:00:06.012642 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 7819
I20250904 20:00:06.013434 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250904 20:00:06.022238 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.5:0
--local_ip_for_outbound_sockets=127.2.182.5
--webserver_interface=127.2.182.5
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--builtin_ntp_servers=127.2.182.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250904 20:00:06.035627 7949 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38291
I20250904 20:00:06.036075 7949 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:06.037168 7949 heartbeater.cc:507] Master 127.2.182.62:38291 requested a full tablet report, sending...
I20250904 20:00:06.039628 7361 ts_manager.cc:194] Registered new tserver with Master: 5ba1e6971c6c42a19e7b5c96ef0665e5 (127.2.182.4:45799)
I20250904 20:00:06.040958 7361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.4:35763
W20250904 20:00:06.342531 7953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:06.343093 7953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:06.343598 7953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:06.375507 7953 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250904 20:00:06.375931 7953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:06.376749 7953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.5
I20250904 20:00:06.413307 7953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.2.182.5
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.5
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:06.414752 7953 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:06.416528 7953 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:06.429217 7960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:07.044595 7949 heartbeater.cc:499] Master 127.2.182.62:38291 was elected leader, sending a full tablet report...
W20250904 20:00:08.003702 7953 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.574s user 0.567s sys 0.992s
W20250904 20:00:07.833040 7959 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7953
W20250904 20:00:06.429811 7961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:08.004866 7953 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.576s user 0.567s sys 0.993s
W20250904 20:00:08.006385 7963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:08.010259 7962 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1574 milliseconds
I20250904 20:00:08.010413 7953 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:08.012400 7953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:08.015205 7953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:08.016790 7953 hybrid_clock.cc:648] HybridClock initialized: now 1757016008016748 us; error 38 us; skew 500 ppm
I20250904 20:00:08.017971 7953 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:08.025200 7953 webserver.cc:480] Webserver started at http://127.2.182.5:44127/ using document root <none> and password file <none>
I20250904 20:00:08.026219 7953 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:08.026436 7953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:08.026929 7953 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:08.031716 7953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "cfe6d59e464948eba185a4542232594f"
format_stamp: "Formatted at 2025-09-04 20:00:08 on dist-test-slave-tjsx"
I20250904 20:00:08.032999 7953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "cfe6d59e464948eba185a4542232594f"
format_stamp: "Formatted at 2025-09-04 20:00:08 on dist-test-slave-tjsx"
I20250904 20:00:08.040647 7953 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250904 20:00:08.048282 7970 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:08.049777 7953 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250904 20:00:08.050362 7953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cfe6d59e464948eba185a4542232594f"
format_stamp: "Formatted at 2025-09-04 20:00:08 on dist-test-slave-tjsx"
I20250904 20:00:08.050832 7953 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:08.106559 7953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:08.108237 7953 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:08.108752 7953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:08.111434 7953 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:08.116051 7953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:08.116302 7953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:08.116596 7953 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:08.116770 7953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:08.267861 7953 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.5:35765
I20250904 20:00:08.268061 8083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.5:35765 every 8 connection(s)
I20250904 20:00:08.270534 7953 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250904 20:00:08.278363 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 7953
I20250904 20:00:08.278928 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250904 20:00:08.294580 8084 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38291
I20250904 20:00:08.295149 8084 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:08.296366 8084 heartbeater.cc:507] Master 127.2.182.62:38291 requested a full tablet report, sending...
I20250904 20:00:08.298926 7360 ts_manager.cc:194] Registered new tserver with Master: cfe6d59e464948eba185a4542232594f (127.2.182.5:35765)
I20250904 20:00:08.300554 7360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.5:43375
I20250904 20:00:08.301187 2776 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250904 20:00:08.340790 7360 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47222:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250904 20:00:08.424048 8018 tablet_service.cc:1468] Processing CreateTablet for tablet 84c679e1e91149bca0e174d9e79103c4 (DEFAULT_TABLE table=TestTable [id=d9495c2b2dce474d948825478e0b7c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:08.426292 8018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84c679e1e91149bca0e174d9e79103c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:08.433879 7617 tablet_service.cc:1468] Processing CreateTablet for tablet 84c679e1e91149bca0e174d9e79103c4 (DEFAULT_TABLE table=TestTable [id=d9495c2b2dce474d948825478e0b7c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:08.433884 7750 tablet_service.cc:1468] Processing CreateTablet for tablet 84c679e1e91149bca0e174d9e79103c4 (DEFAULT_TABLE table=TestTable [id=d9495c2b2dce474d948825478e0b7c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:08.436023 7750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84c679e1e91149bca0e174d9e79103c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:08.436156 7617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84c679e1e91149bca0e174d9e79103c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:08.457904 8103 tablet_bootstrap.cc:492] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f: Bootstrap starting.
I20250904 20:00:08.460412 8104 tablet_bootstrap.cc:492] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee: Bootstrap starting.
I20250904 20:00:08.464607 8105 tablet_bootstrap.cc:492] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a: Bootstrap starting.
I20250904 20:00:08.467079 8103 tablet_bootstrap.cc:654] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:08.468520 8104 tablet_bootstrap.cc:654] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:08.469810 8103 log.cc:826] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:08.471122 8104 log.cc:826] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:08.472169 8105 tablet_bootstrap.cc:654] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:08.474476 8105 log.cc:826] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:08.476007 8103 tablet_bootstrap.cc:492] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f: No bootstrap required, opened a new log
I20250904 20:00:08.476605 8103 ts_tablet_manager.cc:1397] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20250904 20:00:08.478799 8104 tablet_bootstrap.cc:492] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee: No bootstrap required, opened a new log
I20250904 20:00:08.479346 8104 ts_tablet_manager.cc:1397] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20250904 20:00:08.482322 8105 tablet_bootstrap.cc:492] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a: No bootstrap required, opened a new log
I20250904 20:00:08.483014 8105 ts_tablet_manager.cc:1397] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.005s
I20250904 20:00:08.504815 8103 raft_consensus.cc:357] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.505895 8103 raft_consensus.cc:383] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:08.506234 8103 raft_consensus.cc:738] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfe6d59e464948eba185a4542232594f, State: Initialized, Role: FOLLOWER
I20250904 20:00:08.505961 8105 raft_consensus.cc:357] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.506789 8105 raft_consensus.cc:383] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:08.507134 8105 raft_consensus.cc:738] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee7037a4c6247db83f6a139b9b8610a, State: Initialized, Role: FOLLOWER
I20250904 20:00:08.507239 8103 consensus_queue.cc:260] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.507918 8105 consensus_queue.cc:260] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.508518 8104 raft_consensus.cc:357] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.509541 8104 raft_consensus.cc:383] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:08.509879 8104 raft_consensus.cc:738] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d16bacd0f64f5aaa585f5a63083dee, State: Initialized, Role: FOLLOWER
I20250904 20:00:08.510960 8104 consensus_queue.cc:260] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.518539 8104 ts_tablet_manager.cc:1428] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee: Time spent starting tablet: real 0.039s user 0.034s sys 0.004s
I20250904 20:00:08.518720 8103 ts_tablet_manager.cc:1428] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f: Time spent starting tablet: real 0.042s user 0.029s sys 0.012s
I20250904 20:00:08.519394 8084 heartbeater.cc:499] Master 127.2.182.62:38291 was elected leader, sending a full tablet report...
I20250904 20:00:08.519853 8105 ts_tablet_manager.cc:1428] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a: Time spent starting tablet: real 0.036s user 0.017s sys 0.016s
W20250904 20:00:08.526314 8085 tablet.cc:2378] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 20:00:08.668617 7683 tablet.cc:2378] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 20:00:08.712998 8111 raft_consensus.cc:491] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 20:00:08.713487 8111 raft_consensus.cc:513] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.716099 8111 leader_election.cc:290] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfe6d59e464948eba185a4542232594f (127.2.182.5:35765), eee7037a4c6247db83f6a139b9b8610a (127.2.182.3:42757)
I20250904 20:00:08.728996 8038 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84c679e1e91149bca0e174d9e79103c4" candidate_uuid: "12d16bacd0f64f5aaa585f5a63083dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe6d59e464948eba185a4542232594f" is_pre_election: true
I20250904 20:00:08.729288 7770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84c679e1e91149bca0e174d9e79103c4" candidate_uuid: "12d16bacd0f64f5aaa585f5a63083dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee7037a4c6247db83f6a139b9b8610a" is_pre_election: true
I20250904 20:00:08.729765 8038 raft_consensus.cc:2466] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d16bacd0f64f5aaa585f5a63083dee in term 0.
I20250904 20:00:08.730005 7770 raft_consensus.cc:2466] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d16bacd0f64f5aaa585f5a63083dee in term 0.
I20250904 20:00:08.731117 7573 leader_election.cc:304] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d16bacd0f64f5aaa585f5a63083dee, cfe6d59e464948eba185a4542232594f; no voters:
I20250904 20:00:08.731851 8111 raft_consensus.cc:2802] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 20:00:08.732138 8111 raft_consensus.cc:491] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 20:00:08.732385 8111 raft_consensus.cc:3058] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:08.737044 8111 raft_consensus.cc:513] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.738466 8111 leader_election.cc:290] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [CANDIDATE]: Term 1 election: Requested vote from peers cfe6d59e464948eba185a4542232594f (127.2.182.5:35765), eee7037a4c6247db83f6a139b9b8610a (127.2.182.3:42757)
I20250904 20:00:08.739467 8038 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84c679e1e91149bca0e174d9e79103c4" candidate_uuid: "12d16bacd0f64f5aaa585f5a63083dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfe6d59e464948eba185a4542232594f"
I20250904 20:00:08.739570 7770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84c679e1e91149bca0e174d9e79103c4" candidate_uuid: "12d16bacd0f64f5aaa585f5a63083dee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee7037a4c6247db83f6a139b9b8610a"
W20250904 20:00:08.739719 7816 tablet.cc:2378] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 20:00:08.740044 8038 raft_consensus.cc:3058] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:08.740077 7770 raft_consensus.cc:3058] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:08.746956 8038 raft_consensus.cc:2466] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d16bacd0f64f5aaa585f5a63083dee in term 1.
I20250904 20:00:08.746955 7770 raft_consensus.cc:2466] T 84c679e1e91149bca0e174d9e79103c4 P eee7037a4c6247db83f6a139b9b8610a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d16bacd0f64f5aaa585f5a63083dee in term 1.
I20250904 20:00:08.748087 7573 leader_election.cc:304] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d16bacd0f64f5aaa585f5a63083dee, cfe6d59e464948eba185a4542232594f; no voters:
I20250904 20:00:08.748940 8111 raft_consensus.cc:2802] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 1 FOLLOWER]: Leader election won for term 1
I20250904 20:00:08.750608 8111 raft_consensus.cc:695] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [term 1 LEADER]: Becoming Leader. State: Replica: 12d16bacd0f64f5aaa585f5a63083dee, State: Running, Role: LEADER
I20250904 20:00:08.751493 8111 consensus_queue.cc:237] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } }
I20250904 20:00:08.760758 7361 catalog_manager.cc:5582] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d16bacd0f64f5aaa585f5a63083dee (127.2.182.2). New cstate: current_term: 1 leader_uuid: "12d16bacd0f64f5aaa585f5a63083dee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d16bacd0f64f5aaa585f5a63083dee" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 45101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eee7037a4c6247db83f6a139b9b8610a" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 42757 } health_report { overall_health: UNKNOWN } } }
I20250904 20:00:08.807695 2776 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250904 20:00:08.811609 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 12d16bacd0f64f5aaa585f5a63083dee to finish bootstrapping
I20250904 20:00:08.825307 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver eee7037a4c6247db83f6a139b9b8610a to finish bootstrapping
I20250904 20:00:08.836813 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cfe6d59e464948eba185a4542232594f to finish bootstrapping
I20250904 20:00:08.849030 2776 test_util.cc:276] Using random seed: 722630249
I20250904 20:00:08.875875 2776 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250904 20:00:08.876893 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 7686
W20250904 20:00:08.912971 7571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.3:42757: connect: Connection refused (error 111)
I20250904 20:00:08.913530 8038 raft_consensus.cc:1273] T 84c679e1e91149bca0e174d9e79103c4 P cfe6d59e464948eba185a4542232594f [term 1 FOLLOWER]: Refusing update from remote peer 12d16bacd0f64f5aaa585f5a63083dee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 20:00:08.915561 8111 consensus_queue.cc:1035] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfe6d59e464948eba185a4542232594f" member_type: VOTER last_known_addr { host: "127.2.182.5" port: 35765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250904 20:00:08.917835 7571 consensus_peers.cc:489] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee -> Peer eee7037a4c6247db83f6a139b9b8610a (127.2.182.3:42757): Couldn't send request to peer eee7037a4c6247db83f6a139b9b8610a. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.3:42757: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250904 20:00:08.939043 8128 mvcc.cc:204] Tried to move back new op lower bound from 7196737572490256384 to 7196737571853934592. Current Snapshot: MvccSnapshot[applied={T|T < 7196737572490256384}]
W20250904 20:00:08.984447 7678 debug-util.cc:398] Leaking SignalData structure 0x7b08000acfa0 after lost signal to thread 7554
W20250904 20:00:11.400970 7571 consensus_peers.cc:489] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee -> Peer eee7037a4c6247db83f6a139b9b8610a (127.2.182.3:42757): Couldn't send request to peer eee7037a4c6247db83f6a139b9b8610a. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.3:42757: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250904 20:00:11.865634 7617 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250904 20:00:11.906383 7484 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:00:11.909209 8018 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:00:11.957504 7884 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:00:13.709126 7484 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:00:13.711941 7617 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250904 20:00:13.712033 7884 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:00:13.752903 8018 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250904 20:00:14.003257 7571 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.3:42757: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250904 20:00:14.016633 7571 consensus_peers.cc:489] T 84c679e1e91149bca0e174d9e79103c4 P 12d16bacd0f64f5aaa585f5a63083dee -> Peer eee7037a4c6247db83f6a139b9b8610a (127.2.182.3:42757): Couldn't send request to peer eee7037a4c6247db83f6a139b9b8610a. Status: Network error: Client connection negotiation failed: client connection to 127.2.182.3:42757: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250904 20:00:16.250315 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 7420
I20250904 20:00:16.272130 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 7553
I20250904 20:00:16.311576 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 7819
I20250904 20:00:16.332486 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 7953
I20250904 20:00:16.365975 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 7328
2025-09-04T20:00:16Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (20641 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (20641 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250904 20:00:16.427362 2776 test_util.cc:276] Using random seed: 730208582
I20250904 20:00:16.431977 2776 ts_itest-base.cc:115] Starting cluster with:
I20250904 20:00:16.432160 2776 ts_itest-base.cc:116] --------------
I20250904 20:00:16.432284 2776 ts_itest-base.cc:117] 1 tablet servers
I20250904 20:00:16.432394 2776 ts_itest-base.cc:118] 1 replicas per TS
I20250904 20:00:16.432498 2776 ts_itest-base.cc:119] --------------
2025-09-04T20:00:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T20:00:16Z Disabled control of system clock
I20250904 20:00:16.475929 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41345
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:41345 with env {}
W20250904 20:00:16.782426 8231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:16.783079 8231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:16.783552 8231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:16.815312 8231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 20:00:16.815688 8231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:16.815937 8231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 20:00:16.816175 8231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 20:00:16.855773 8231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:41345
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:41345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:16.857481 8231 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:16.859565 8231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:16.873337 8237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:18.275121 8236 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 8231
W20250904 20:00:16.873930 8238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:18.458913 8239 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1584 milliseconds
W20250904 20:00:18.458071 8231 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.585s user 0.619s sys 0.956s
W20250904 20:00:18.460131 8240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:18.460522 8231 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.588s user 0.620s sys 0.956s
I20250904 20:00:18.461030 8231 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:18.465447 8231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:18.468094 8231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:18.469516 8231 hybrid_clock.cc:648] HybridClock initialized: now 1757016018469484 us; error 38 us; skew 500 ppm
I20250904 20:00:18.470371 8231 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:18.477121 8231 webserver.cc:480] Webserver started at http://127.2.182.62:43879/ using document root <none> and password file <none>
I20250904 20:00:18.478207 8231 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:18.478466 8231 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:18.479032 8231 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:18.483732 8231 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9a203bff6b594e6aa7b71995b512fc9f"
format_stamp: "Formatted at 2025-09-04 20:00:18 on dist-test-slave-tjsx"
I20250904 20:00:18.484967 8231 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9a203bff6b594e6aa7b71995b512fc9f"
format_stamp: "Formatted at 2025-09-04 20:00:18 on dist-test-slave-tjsx"
I20250904 20:00:18.492961 8231 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250904 20:00:18.499140 8247 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:18.500384 8231 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250904 20:00:18.500728 8231 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9a203bff6b594e6aa7b71995b512fc9f"
format_stamp: "Formatted at 2025-09-04 20:00:18 on dist-test-slave-tjsx"
I20250904 20:00:18.501058 8231 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:18.548527 8231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:18.550096 8231 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:18.550558 8231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:18.623306 8231 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:41345
I20250904 20:00:18.623420 8298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:41345 every 8 connection(s)
I20250904 20:00:18.626204 8231 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250904 20:00:18.626868 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 8231
I20250904 20:00:18.627291 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250904 20:00:18.633401 8299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:18.660458 8299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f: Bootstrap starting.
I20250904 20:00:18.667032 8299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:18.669047 8299 log.cc:826] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:18.674034 8299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f: No bootstrap required, opened a new log
I20250904 20:00:18.692505 8299 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a203bff6b594e6aa7b71995b512fc9f" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41345 } }
I20250904 20:00:18.693301 8299 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:18.693560 8299 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a203bff6b594e6aa7b71995b512fc9f, State: Initialized, Role: FOLLOWER
I20250904 20:00:18.694379 8299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a203bff6b594e6aa7b71995b512fc9f" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41345 } }
I20250904 20:00:18.695125 8299 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 20:00:18.695492 8299 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 20:00:18.695889 8299 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:18.700722 8299 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a203bff6b594e6aa7b71995b512fc9f" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41345 } }
I20250904 20:00:18.701557 8299 leader_election.cc:304] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a203bff6b594e6aa7b71995b512fc9f; no voters:
I20250904 20:00:18.703531 8299 leader_election.cc:290] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 20:00:18.704257 8304 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 1 FOLLOWER]: Leader election won for term 1
I20250904 20:00:18.706595 8304 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [term 1 LEADER]: Becoming Leader. State: Replica: 9a203bff6b594e6aa7b71995b512fc9f, State: Running, Role: LEADER
I20250904 20:00:18.707502 8304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a203bff6b594e6aa7b71995b512fc9f" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41345 } }
I20250904 20:00:18.708603 8299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [sys.catalog]: configured and running, proceeding with master startup.
I20250904 20:00:18.717262 8306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a203bff6b594e6aa7b71995b512fc9f. Latest consensus state: current_term: 1 leader_uuid: "9a203bff6b594e6aa7b71995b512fc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a203bff6b594e6aa7b71995b512fc9f" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41345 } } }
I20250904 20:00:18.718137 8306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [sys.catalog]: This master's current role is: LEADER
I20250904 20:00:18.719460 8305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a203bff6b594e6aa7b71995b512fc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a203bff6b594e6aa7b71995b512fc9f" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 41345 } } }
I20250904 20:00:18.720284 8305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f [sys.catalog]: This master's current role is: LEADER
I20250904 20:00:18.723147 8312 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 20:00:18.735522 8312 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 20:00:18.753991 8312 catalog_manager.cc:1349] Generated new cluster ID: 37cd86fbfa714204b3fe69ba399efccd
I20250904 20:00:18.754335 8312 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 20:00:18.774322 8312 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 20:00:18.776264 8312 catalog_manager.cc:1506] Loading token signing keys...
I20250904 20:00:18.791772 8312 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9a203bff6b594e6aa7b71995b512fc9f: Generated new TSK 0
I20250904 20:00:18.792861 8312 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 20:00:18.815584 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:41345
--builtin_ntp_servers=127.2.182.20:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250904 20:00:19.167040 8323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:19.167745 8323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:19.168282 8323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:19.203401 8323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:19.204404 8323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 20:00:19.244882 8323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:41345
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:19.246591 8323 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:19.248675 8323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:19.266577 8330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:19.266614 8329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:20.553843 8323 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.290s user 0.523s sys 0.760s
W20250904 20:00:20.554584 8323 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.291s user 0.523s sys 0.761s
I20250904 20:00:20.559544 8323 server_base.cc:1047] running on GCE node
W20250904 20:00:20.561479 8334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:20.562849 8323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:20.569204 8323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:20.570796 8323 hybrid_clock.cc:648] HybridClock initialized: now 1757016020570706 us; error 106 us; skew 500 ppm
I20250904 20:00:20.571802 8323 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:20.579921 8323 webserver.cc:480] Webserver started at http://127.2.182.1:42199/ using document root <none> and password file <none>
I20250904 20:00:20.580992 8323 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:20.581239 8323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:20.581740 8323 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:20.586817 8323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ce6aa6ae5234485ab2828527465d680c"
format_stamp: "Formatted at 2025-09-04 20:00:20 on dist-test-slave-tjsx"
I20250904 20:00:20.588086 8323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ce6aa6ae5234485ab2828527465d680c"
format_stamp: "Formatted at 2025-09-04 20:00:20 on dist-test-slave-tjsx"
I20250904 20:00:20.597368 8323 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.002s
I20250904 20:00:20.604526 8339 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:20.605858 8323 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250904 20:00:20.606232 8323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ce6aa6ae5234485ab2828527465d680c"
format_stamp: "Formatted at 2025-09-04 20:00:20 on dist-test-slave-tjsx"
I20250904 20:00:20.606618 8323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:20.697862 8323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:20.699502 8323 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:20.699980 8323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:20.702732 8323 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:20.707674 8323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:20.707909 8323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:20.708170 8323 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:20.708339 8323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:20.886881 8323 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:41319
I20250904 20:00:20.886998 8451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:41319 every 8 connection(s)
I20250904 20:00:20.889748 8323 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250904 20:00:20.898793 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 8323
I20250904 20:00:20.899334 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757015875386078-2776-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250904 20:00:20.918597 8452 heartbeater.cc:344] Connected to a master server at 127.2.182.62:41345
I20250904 20:00:20.919128 8452 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:20.920244 8452 heartbeater.cc:507] Master 127.2.182.62:41345 requested a full tablet report, sending...
I20250904 20:00:20.923087 8264 ts_manager.cc:194] Registered new tserver with Master: ce6aa6ae5234485ab2828527465d680c (127.2.182.1:41319)
I20250904 20:00:20.925073 8264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:49951
I20250904 20:00:20.933120 2776 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250904 20:00:20.964511 8264 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39498:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250904 20:00:21.024920 8387 tablet_service.cc:1468] Processing CreateTablet for tablet ccdce6e74319431d8ca9784612174534 (DEFAULT_TABLE table=TestTable [id=a86ba30392234c6ca7c8f3f1d1077b4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:21.026849 8387 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccdce6e74319431d8ca9784612174534. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:21.048842 8467 tablet_bootstrap.cc:492] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c: Bootstrap starting.
I20250904 20:00:21.054641 8467 tablet_bootstrap.cc:654] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:21.056770 8467 log.cc:826] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:21.062177 8467 tablet_bootstrap.cc:492] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c: No bootstrap required, opened a new log
I20250904 20:00:21.062638 8467 ts_tablet_manager.cc:1397] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20250904 20:00:21.081251 8467 raft_consensus.cc:357] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6aa6ae5234485ab2828527465d680c" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41319 } }
I20250904 20:00:21.081853 8467 raft_consensus.cc:383] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:21.082046 8467 raft_consensus.cc:738] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6aa6ae5234485ab2828527465d680c, State: Initialized, Role: FOLLOWER
I20250904 20:00:21.082809 8467 consensus_queue.cc:260] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6aa6ae5234485ab2828527465d680c" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41319 } }
I20250904 20:00:21.083343 8467 raft_consensus.cc:397] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 20:00:21.083597 8467 raft_consensus.cc:491] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 20:00:21.083925 8467 raft_consensus.cc:3058] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:21.088843 8467 raft_consensus.cc:513] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6aa6ae5234485ab2828527465d680c" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41319 } }
I20250904 20:00:21.089627 8467 leader_election.cc:304] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce6aa6ae5234485ab2828527465d680c; no voters:
I20250904 20:00:21.091361 8467 leader_election.cc:290] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 20:00:21.091779 8469 raft_consensus.cc:2802] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 1 FOLLOWER]: Leader election won for term 1
I20250904 20:00:21.094643 8469 raft_consensus.cc:695] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [term 1 LEADER]: Becoming Leader. State: Replica: ce6aa6ae5234485ab2828527465d680c, State: Running, Role: LEADER
I20250904 20:00:21.095316 8452 heartbeater.cc:499] Master 127.2.182.62:41345 was elected leader, sending a full tablet report...
I20250904 20:00:21.095708 8469 consensus_queue.cc:237] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6aa6ae5234485ab2828527465d680c" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41319 } }
I20250904 20:00:21.096932 8467 ts_tablet_manager.cc:1428] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c: Time spent starting tablet: real 0.034s user 0.028s sys 0.004s
I20250904 20:00:21.107357 8264 catalog_manager.cc:5582] T ccdce6e74319431d8ca9784612174534 P ce6aa6ae5234485ab2828527465d680c reported cstate change: term changed from 0 to 1, leader changed from <none> to ce6aa6ae5234485ab2828527465d680c (127.2.182.1). New cstate: current_term: 1 leader_uuid: "ce6aa6ae5234485ab2828527465d680c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce6aa6ae5234485ab2828527465d680c" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 41319 } health_report { overall_health: HEALTHY } } }
I20250904 20:00:21.134074 2776 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250904 20:00:21.138124 2776 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ce6aa6ae5234485ab2828527465d680c to finish bootstrapping
I20250904 20:00:23.761571 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 8323
I20250904 20:00:23.788769 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 8231
2025-09-04T20:00:23Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7419 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7419 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250904 20:00:23.847136 2776 test_util.cc:276] Using random seed: 737628355
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (13 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-09-04T20:00:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-04T20:00:23Z Disabled control of system clock
I20250904 20:00:23.900883 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38749
--webserver_interface=127.2.182.62
--webserver_port=0
--builtin_ntp_servers=127.2.182.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38749 with env {}
W20250904 20:00:24.228801 8495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:24.229424 8495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:24.229897 8495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:24.262456 8495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 20:00:24.262866 8495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:24.263130 8495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 20:00:24.263381 8495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 20:00:24.300344 8495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38749
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:24.301688 8495 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:24.303453 8495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:24.315636 8501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:25.718384 8500 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 8495
W20250904 20:00:25.904474 8500 kernel_stack_watchdog.cc:198] Thread 8495 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 20:00:24.316854 8502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:25.905462 8495 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.590s user 0.000s sys 0.002s
W20250904 20:00:25.905818 8495 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.591s user 0.000s sys 0.003s
W20250904 20:00:25.905998 8503 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1588 milliseconds
W20250904 20:00:25.908015 8504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:25.907961 8495 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:25.910382 8495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:25.913002 8495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:25.914407 8495 hybrid_clock.cc:648] HybridClock initialized: now 1757016025914373 us; error 41 us; skew 500 ppm
I20250904 20:00:25.915298 8495 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:25.921805 8495 webserver.cc:480] Webserver started at http://127.2.182.62:38585/ using document root <none> and password file <none>
I20250904 20:00:25.922894 8495 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:25.923142 8495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:25.923627 8495 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:25.928364 8495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/instance:
uuid: "85fc16f2a7fe411d81562d1b0fdfb137"
format_stamp: "Formatted at 2025-09-04 20:00:25 on dist-test-slave-tjsx"
I20250904 20:00:25.929644 8495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal/instance:
uuid: "85fc16f2a7fe411d81562d1b0fdfb137"
format_stamp: "Formatted at 2025-09-04 20:00:25 on dist-test-slave-tjsx"
I20250904 20:00:25.937494 8495 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250904 20:00:25.943629 8512 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:25.944813 8495 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250904 20:00:25.945216 8495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
uuid: "85fc16f2a7fe411d81562d1b0fdfb137"
format_stamp: "Formatted at 2025-09-04 20:00:25 on dist-test-slave-tjsx"
I20250904 20:00:25.945585 8495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:25.994326 8495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:25.995913 8495 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:25.996402 8495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:26.069063 8495 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38749
I20250904 20:00:26.069129 8563 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:38749 every 8 connection(s)
I20250904 20:00:26.072016 8495 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/info.pb
I20250904 20:00:26.073108 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 8495
I20250904 20:00:26.073696 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal/instance
I20250904 20:00:26.078944 8564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:26.101850 8564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137: Bootstrap starting.
I20250904 20:00:26.108376 8564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:26.110244 8564 log.cc:826] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:26.115195 8564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137: No bootstrap required, opened a new log
I20250904 20:00:26.134110 8564 raft_consensus.cc:357] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fc16f2a7fe411d81562d1b0fdfb137" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } }
I20250904 20:00:26.134876 8564 raft_consensus.cc:383] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:26.135134 8564 raft_consensus.cc:738] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85fc16f2a7fe411d81562d1b0fdfb137, State: Initialized, Role: FOLLOWER
I20250904 20:00:26.135934 8564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fc16f2a7fe411d81562d1b0fdfb137" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } }
I20250904 20:00:26.136539 8564 raft_consensus.cc:397] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 20:00:26.136821 8564 raft_consensus.cc:491] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 20:00:26.137136 8564 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:26.141536 8564 raft_consensus.cc:513] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fc16f2a7fe411d81562d1b0fdfb137" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } }
I20250904 20:00:26.142333 8564 leader_election.cc:304] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85fc16f2a7fe411d81562d1b0fdfb137; no voters:
I20250904 20:00:26.144249 8564 leader_election.cc:290] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 20:00:26.144937 8569 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 20:00:26.147295 8569 raft_consensus.cc:695] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [term 1 LEADER]: Becoming Leader. State: Replica: 85fc16f2a7fe411d81562d1b0fdfb137, State: Running, Role: LEADER
I20250904 20:00:26.148134 8569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fc16f2a7fe411d81562d1b0fdfb137" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } }
I20250904 20:00:26.149197 8564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 20:00:26.157229 8570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85fc16f2a7fe411d81562d1b0fdfb137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fc16f2a7fe411d81562d1b0fdfb137" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } } }
I20250904 20:00:26.158744 8571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85fc16f2a7fe411d81562d1b0fdfb137. Latest consensus state: current_term: 1 leader_uuid: "85fc16f2a7fe411d81562d1b0fdfb137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85fc16f2a7fe411d81562d1b0fdfb137" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } } }
I20250904 20:00:26.159549 8571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [sys.catalog]: This master's current role is: LEADER
I20250904 20:00:26.160985 8570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137 [sys.catalog]: This master's current role is: LEADER
I20250904 20:00:26.164561 8578 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 20:00:26.177690 8578 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 20:00:26.194995 8578 catalog_manager.cc:1349] Generated new cluster ID: ffa8d22f5ba44dac8653e7128d9262ed
I20250904 20:00:26.195354 8578 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 20:00:26.210512 8578 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 20:00:26.212882 8578 catalog_manager.cc:1506] Loading token signing keys...
I20250904 20:00:26.229142 8578 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 85fc16f2a7fe411d81562d1b0fdfb137: Generated new TSK 0
I20250904 20:00:26.230445 8578 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250904 20:00:26.245962 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:0
--local_ip_for_outbound_sockets=127.2.182.1
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38749
--builtin_ntp_servers=127.2.182.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250904 20:00:26.570026 8588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:26.570571 8588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:26.571115 8588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:26.603449 8588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:26.604354 8588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 20:00:26.641981 8588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:26.643448 8588 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:26.645246 8588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:26.658814 8594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:28.062477 8593 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 8588
W20250904 20:00:28.373726 8588 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.713s user 0.000s sys 0.002s
W20250904 20:00:28.374770 8588 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.714s user 0.001s sys 0.002s
W20250904 20:00:28.374666 8596 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1714 milliseconds
W20250904 20:00:26.659747 8595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:28.376376 8598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:28.376406 8588 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:28.380560 8588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:28.392027 8588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:28.393602 8588 hybrid_clock.cc:648] HybridClock initialized: now 1757016028393514 us; error 81 us; skew 500 ppm
I20250904 20:00:28.394551 8588 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:28.401687 8588 webserver.cc:480] Webserver started at http://127.2.182.1:38989/ using document root <none> and password file <none>
I20250904 20:00:28.403005 8588 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:28.403373 8588 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:28.403859 8588 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:28.409881 8588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/instance:
uuid: "f1d4552e31a64a6eb400751dd89f5f79"
format_stamp: "Formatted at 2025-09-04 20:00:28 on dist-test-slave-tjsx"
I20250904 20:00:28.411325 8588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal/instance:
uuid: "f1d4552e31a64a6eb400751dd89f5f79"
format_stamp: "Formatted at 2025-09-04 20:00:28 on dist-test-slave-tjsx"
I20250904 20:00:28.420029 8588 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20250904 20:00:28.426867 8604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:28.428109 8588 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20250904 20:00:28.428467 8588 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
uuid: "f1d4552e31a64a6eb400751dd89f5f79"
format_stamp: "Formatted at 2025-09-04 20:00:28 on dist-test-slave-tjsx"
I20250904 20:00:28.428824 8588 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:28.495534 8588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:28.497288 8588 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:28.497778 8588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:28.500851 8588 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:28.506222 8588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:28.506465 8588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:28.506783 8588 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:28.506953 8588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:28.682399 8588 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:44495
I20250904 20:00:28.682588 8716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:44495 every 8 connection(s)
I20250904 20:00:28.686314 8588 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/info.pb
I20250904 20:00:28.687487 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 8588
I20250904 20:00:28.688007 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal/instance
I20250904 20:00:28.697325 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:0
--local_ip_for_outbound_sockets=127.2.182.2
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38749
--builtin_ntp_servers=127.2.182.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250904 20:00:28.713003 8717 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38749
I20250904 20:00:28.713626 8717 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:28.715243 8717 heartbeater.cc:507] Master 127.2.182.62:38749 requested a full tablet report, sending...
I20250904 20:00:28.718919 8529 ts_manager.cc:194] Registered new tserver with Master: f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1:44495)
I20250904 20:00:28.721994 8529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:39933
W20250904 20:00:29.010825 8721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:29.011304 8721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:29.011837 8721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:29.046737 8721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:29.048121 8721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 20:00:29.100443 8721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:29.102137 8721 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:29.103900 8721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:29.116279 8727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:29.727557 8717 heartbeater.cc:499] Master 127.2.182.62:38749 was elected leader, sending a full tablet report...
W20250904 20:00:29.116827 8728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:30.468863 8729 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1347 milliseconds
W20250904 20:00:30.469565 8721 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.353s user 0.524s sys 0.823s
W20250904 20:00:30.469929 8721 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.353s user 0.524s sys 0.823s
W20250904 20:00:30.470196 8730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:30.470223 8721 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:30.471868 8721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:30.474606 8721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:30.476182 8721 hybrid_clock.cc:648] HybridClock initialized: now 1757016030476137 us; error 43 us; skew 500 ppm
I20250904 20:00:30.477380 8721 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:30.490909 8721 webserver.cc:480] Webserver started at http://127.2.182.2:44355/ using document root <none> and password file <none>
I20250904 20:00:30.492270 8721 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:30.492578 8721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:30.493177 8721 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:30.500124 8721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/instance:
uuid: "4184c1efc269478880db76e7429daf9b"
format_stamp: "Formatted at 2025-09-04 20:00:30 on dist-test-slave-tjsx"
I20250904 20:00:30.501909 8721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal/instance:
uuid: "4184c1efc269478880db76e7429daf9b"
format_stamp: "Formatted at 2025-09-04 20:00:30 on dist-test-slave-tjsx"
I20250904 20:00:30.512403 8721 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s
I20250904 20:00:30.520960 8737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:30.522430 8721 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250904 20:00:30.522948 8721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
uuid: "4184c1efc269478880db76e7429daf9b"
format_stamp: "Formatted at 2025-09-04 20:00:30 on dist-test-slave-tjsx"
I20250904 20:00:30.523447 8721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:30.599885 8721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:30.601516 8721 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:30.602007 8721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:30.604694 8721 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:30.609236 8721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:30.609510 8721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:30.609791 8721 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:30.609958 8721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:30.752126 8721 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:32953
I20250904 20:00:30.752223 8849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:32953 every 8 connection(s)
I20250904 20:00:30.754858 8721 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/info.pb
I20250904 20:00:30.763033 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 8721
I20250904 20:00:30.763566 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal/instance
I20250904 20:00:30.771059 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:0
--local_ip_for_outbound_sockets=127.2.182.3
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38749
--builtin_ntp_servers=127.2.182.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250904 20:00:30.779685 8850 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38749
I20250904 20:00:30.780134 8850 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:30.781388 8850 heartbeater.cc:507] Master 127.2.182.62:38749 requested a full tablet report, sending...
I20250904 20:00:30.784000 8529 ts_manager.cc:194] Registered new tserver with Master: 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953)
I20250904 20:00:30.785310 8529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:36135
W20250904 20:00:31.100239 8854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:31.100777 8854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:31.101287 8854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:31.133505 8854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:31.134441 8854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 20:00:31.170670 8854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=0
--tserver_master_addrs=127.2.182.62:38749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:31.172066 8854 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:31.173833 8854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:31.187289 8860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:31.789417 8850 heartbeater.cc:499] Master 127.2.182.62:38749 was elected leader, sending a full tablet report...
W20250904 20:00:31.188118 8861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:32.294636 8863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:32.297134 8862 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
I20250904 20:00:32.297281 8854 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:32.298534 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:32.301280 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:32.302790 8854 hybrid_clock.cc:648] HybridClock initialized: now 1757016032302746 us; error 60 us; skew 500 ppm
I20250904 20:00:32.303659 8854 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:32.310413 8854 webserver.cc:480] Webserver started at http://127.2.182.3:45853/ using document root <none> and password file <none>
I20250904 20:00:32.311476 8854 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:32.311705 8854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:32.312182 8854 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:32.318164 8854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/instance:
uuid: "83c54ebbf0154f888587972d9b3187b7"
format_stamp: "Formatted at 2025-09-04 20:00:32 on dist-test-slave-tjsx"
I20250904 20:00:32.319509 8854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal/instance:
uuid: "83c54ebbf0154f888587972d9b3187b7"
format_stamp: "Formatted at 2025-09-04 20:00:32 on dist-test-slave-tjsx"
I20250904 20:00:32.327358 8854 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250904 20:00:32.333513 8870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:32.334769 8854 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250904 20:00:32.335132 8854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
uuid: "83c54ebbf0154f888587972d9b3187b7"
format_stamp: "Formatted at 2025-09-04 20:00:32 on dist-test-slave-tjsx"
I20250904 20:00:32.335505 8854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:32.388847 8854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:32.390352 8854 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:32.390832 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:32.393635 8854 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:32.397939 8854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250904 20:00:32.398170 8854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:32.398437 8854 ts_tablet_manager.cc:610] Registered 0 tablets
I20250904 20:00:32.398590 8854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:32.536936 8854 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:39333
I20250904 20:00:32.537030 8982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:39333 every 8 connection(s)
I20250904 20:00:32.539636 8854 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/info.pb
I20250904 20:00:32.550153 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 8854
I20250904 20:00:32.550611 2776 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal/instance
I20250904 20:00:32.566375 8983 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38749
I20250904 20:00:32.566875 8983 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:32.568215 8983 heartbeater.cc:507] Master 127.2.182.62:38749 requested a full tablet report, sending...
I20250904 20:00:32.571197 8528 ts_manager.cc:194] Registered new tserver with Master: 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333)
I20250904 20:00:32.572660 8528 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:43607
I20250904 20:00:32.587184 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 20:00:32.617538 2776 test_util.cc:276] Using random seed: 746398756
I20250904 20:00:32.658577 8528 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44070:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 20:00:32.661181 8528 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 20:00:32.723551 8652 tablet_service.cc:1468] Processing CreateTablet for tablet ccb113d82f82495fb34b46b81acb8337 (DEFAULT_TABLE table=pre_rebuild [id=afbc8d11c78e4782b02616bfd8896596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:32.723548 8785 tablet_service.cc:1468] Processing CreateTablet for tablet ccb113d82f82495fb34b46b81acb8337 (DEFAULT_TABLE table=pre_rebuild [id=afbc8d11c78e4782b02616bfd8896596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:32.724288 8918 tablet_service.cc:1468] Processing CreateTablet for tablet ccb113d82f82495fb34b46b81acb8337 (DEFAULT_TABLE table=pre_rebuild [id=afbc8d11c78e4782b02616bfd8896596]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:32.725598 8785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb113d82f82495fb34b46b81acb8337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:32.725603 8652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb113d82f82495fb34b46b81acb8337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:32.725926 8918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb113d82f82495fb34b46b81acb8337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:32.750937 9007 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Bootstrap starting.
I20250904 20:00:32.763240 9007 tablet_bootstrap.cc:654] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:32.766358 9007 log.cc:826] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:32.776487 9009 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Bootstrap starting.
I20250904 20:00:32.779554 9008 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Bootstrap starting.
I20250904 20:00:32.784643 9009 tablet_bootstrap.cc:654] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:32.787031 9007 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: No bootstrap required, opened a new log
I20250904 20:00:32.787211 9009 log.cc:826] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:32.787644 9007 ts_tablet_manager.cc:1397] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.013s
I20250904 20:00:32.788547 9008 tablet_bootstrap.cc:654] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:32.791134 9008 log.cc:826] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:32.801158 9009 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: No bootstrap required, opened a new log
I20250904 20:00:32.801755 9009 ts_tablet_manager.cc:1397] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.010s
I20250904 20:00:32.804019 9008 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: No bootstrap required, opened a new log
I20250904 20:00:32.804590 9008 ts_tablet_manager.cc:1397] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.006s
I20250904 20:00:32.810866 9007 raft_consensus.cc:357] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.811622 9007 raft_consensus.cc:383] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:32.811909 9007 raft_consensus.cc:738] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c54ebbf0154f888587972d9b3187b7, State: Initialized, Role: FOLLOWER
I20250904 20:00:32.812724 9007 consensus_queue.cc:260] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.815881 8983 heartbeater.cc:499] Master 127.2.182.62:38749 was elected leader, sending a full tablet report...
I20250904 20:00:32.816881 9007 ts_tablet_manager.cc:1428] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Time spent starting tablet: real 0.029s user 0.019s sys 0.009s
I20250904 20:00:32.831954 9008 raft_consensus.cc:357] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.832283 9009 raft_consensus.cc:357] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.832998 9008 raft_consensus.cc:383] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:32.833117 9009 raft_consensus.cc:383] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:32.833472 9008 raft_consensus.cc:738] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4184c1efc269478880db76e7429daf9b, State: Initialized, Role: FOLLOWER
I20250904 20:00:32.833642 9009 raft_consensus.cc:738] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d4552e31a64a6eb400751dd89f5f79, State: Initialized, Role: FOLLOWER
I20250904 20:00:32.834781 9008 consensus_queue.cc:260] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.834772 9009 consensus_queue.cc:260] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.840055 9009 ts_tablet_manager.cc:1428] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Time spent starting tablet: real 0.038s user 0.032s sys 0.004s
I20250904 20:00:32.840739 9008 ts_tablet_manager.cc:1428] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Time spent starting tablet: real 0.036s user 0.032s sys 0.005s
I20250904 20:00:32.864100 9014 raft_consensus.cc:491] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 20:00:32.864961 9014 raft_consensus.cc:513] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.868090 9014 leader_election.cc:290] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333), f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1:44495)
I20250904 20:00:32.882153 8938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "4184c1efc269478880db76e7429daf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c54ebbf0154f888587972d9b3187b7" is_pre_election: true
I20250904 20:00:32.882421 8672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "4184c1efc269478880db76e7429daf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d4552e31a64a6eb400751dd89f5f79" is_pre_election: true
I20250904 20:00:32.883265 8672 raft_consensus.cc:2466] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4184c1efc269478880db76e7429daf9b in term 0.
I20250904 20:00:32.883117 8938 raft_consensus.cc:2466] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4184c1efc269478880db76e7429daf9b in term 0.
I20250904 20:00:32.884730 8741 leader_election.cc:304] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4184c1efc269478880db76e7429daf9b, f1d4552e31a64a6eb400751dd89f5f79; no voters:
I20250904 20:00:32.885540 9014 raft_consensus.cc:2802] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 20:00:32.885890 9014 raft_consensus.cc:491] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 20:00:32.886117 9014 raft_consensus.cc:3058] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:32.890609 9014 raft_consensus.cc:513] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.892122 9014 leader_election.cc:290] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [CANDIDATE]: Term 1 election: Requested vote from peers 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333), f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1:44495)
I20250904 20:00:32.892998 8938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "4184c1efc269478880db76e7429daf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c54ebbf0154f888587972d9b3187b7"
I20250904 20:00:32.893203 8672 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "4184c1efc269478880db76e7429daf9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d4552e31a64a6eb400751dd89f5f79"
I20250904 20:00:32.893466 8938 raft_consensus.cc:3058] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:32.893674 8672 raft_consensus.cc:3058] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:32.898286 8938 raft_consensus.cc:2466] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4184c1efc269478880db76e7429daf9b in term 1.
I20250904 20:00:32.898324 8672 raft_consensus.cc:2466] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4184c1efc269478880db76e7429daf9b in term 1.
I20250904 20:00:32.899401 8741 leader_election.cc:304] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4184c1efc269478880db76e7429daf9b, f1d4552e31a64a6eb400751dd89f5f79; no voters:
I20250904 20:00:32.900151 9014 raft_consensus.cc:2802] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 1 FOLLOWER]: Leader election won for term 1
I20250904 20:00:32.901793 9014 raft_consensus.cc:695] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 1 LEADER]: Becoming Leader. State: Replica: 4184c1efc269478880db76e7429daf9b, State: Running, Role: LEADER
I20250904 20:00:32.902611 9014 consensus_queue.cc:237] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:32.913425 8528 catalog_manager.cc:5582] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4184c1efc269478880db76e7429daf9b (127.2.182.2). New cstate: current_term: 1 leader_uuid: "4184c1efc269478880db76e7429daf9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } health_report { overall_health: UNKNOWN } } }
W20250904 20:00:32.947206 8718 tablet.cc:2378] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 20:00:33.013521 8851 tablet.cc:2378] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 20:00:33.044473 8984 tablet.cc:2378] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 20:00:33.095912 8672 raft_consensus.cc:1273] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Refusing update from remote peer 4184c1efc269478880db76e7429daf9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 20:00:33.095916 8938 raft_consensus.cc:1273] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 1 FOLLOWER]: Refusing update from remote peer 4184c1efc269478880db76e7429daf9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 20:00:33.097961 9014 consensus_queue.cc:1035] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250904 20:00:33.099758 9018 consensus_queue.cc:1035] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250904 20:00:33.145926 9025 mvcc.cc:204] Tried to move back new op lower bound from 7196737671540826112 to 7196737670776987648. Current Snapshot: MvccSnapshot[applied={T|T < 7196737671540826112}]
I20250904 20:00:33.153115 9027 mvcc.cc:204] Tried to move back new op lower bound from 7196737671540826112 to 7196737670776987648. Current Snapshot: MvccSnapshot[applied={T|T < 7196737671540826112}]
I20250904 20:00:33.157373 9026 mvcc.cc:204] Tried to move back new op lower bound from 7196737671540826112 to 7196737670776987648. Current Snapshot: MvccSnapshot[applied={T|T < 7196737671540826112}]
I20250904 20:00:38.611835 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 8495
W20250904 20:00:39.024256 9063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:39.025207 9063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:39.078675 9063 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250904 20:00:39.251214 8717 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38749 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38749: connect: Connection refused (error 111)
W20250904 20:00:39.314954 8850 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38749 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38749: connect: Connection refused (error 111)
W20250904 20:00:39.327837 8983 heartbeater.cc:646] Failed to heartbeat to 127.2.182.62:38749 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.182.62:38749: connect: Connection refused (error 111)
W20250904 20:00:40.328143 9063 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.205s user 0.456s sys 0.741s
W20250904 20:00:40.329267 9063 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.206s user 0.457s sys 0.742s
I20250904 20:00:40.479960 9063 minidump.cc:252] Setting minidump size limit to 20M
I20250904 20:00:40.482769 9063 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:40.484093 9063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:40.496229 9096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:40.500663 9099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:40.498868 9097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:40.582505 9063 server_base.cc:1047] running on GCE node
I20250904 20:00:40.583756 9063 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250904 20:00:40.584321 9063 hybrid_clock.cc:648] HybridClock initialized: now 1757016040584286 us; error 47094 us; skew 500 ppm
I20250904 20:00:40.585405 9063 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:40.590373 9063 webserver.cc:480] Webserver started at http://0.0.0.0:40721/ using document root <none> and password file <none>
I20250904 20:00:40.591353 9063 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:40.591614 9063 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:40.592059 9063 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250904 20:00:40.596666 9063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/instance:
uuid: "307718c8edd342bb9dda5a72c84a6630"
format_stamp: "Formatted at 2025-09-04 20:00:40 on dist-test-slave-tjsx"
I20250904 20:00:40.597918 9063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal/instance:
uuid: "307718c8edd342bb9dda5a72c84a6630"
format_stamp: "Formatted at 2025-09-04 20:00:40 on dist-test-slave-tjsx"
I20250904 20:00:40.604761 9063 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250904 20:00:40.610141 9104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:40.611168 9063 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250904 20:00:40.611539 9063 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
uuid: "307718c8edd342bb9dda5a72c84a6630"
format_stamp: "Formatted at 2025-09-04 20:00:40 on dist-test-slave-tjsx"
I20250904 20:00:40.611872 9063 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:40.670478 9063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:40.672261 9063 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:40.672824 9063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:40.678098 9063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:40.698848 9063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Bootstrap starting.
I20250904 20:00:40.706168 9063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:40.708644 9063 log.cc:826] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:40.714537 9063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: No bootstrap required, opened a new log
I20250904 20:00:40.737567 9063 raft_consensus.cc:357] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER }
I20250904 20:00:40.738098 9063 raft_consensus.cc:383] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:40.738307 9063 raft_consensus.cc:738] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307718c8edd342bb9dda5a72c84a6630, State: Initialized, Role: FOLLOWER
I20250904 20:00:40.739050 9063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER }
I20250904 20:00:40.739550 9063 raft_consensus.cc:397] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 20:00:40.739785 9063 raft_consensus.cc:491] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 20:00:40.740051 9063 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:40.744277 9063 raft_consensus.cc:513] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER }
I20250904 20:00:40.745007 9063 leader_election.cc:304] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307718c8edd342bb9dda5a72c84a6630; no voters:
I20250904 20:00:40.746724 9063 leader_election.cc:290] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250904 20:00:40.747036 9111 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 20:00:40.749665 9111 raft_consensus.cc:695] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 LEADER]: Becoming Leader. State: Replica: 307718c8edd342bb9dda5a72c84a6630, State: Running, Role: LEADER
I20250904 20:00:40.750624 9111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER }
I20250904 20:00:40.758239 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307718c8edd342bb9dda5a72c84a6630. Latest consensus state: current_term: 1 leader_uuid: "307718c8edd342bb9dda5a72c84a6630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER } }
I20250904 20:00:40.758533 9112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "307718c8edd342bb9dda5a72c84a6630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER } }
I20250904 20:00:40.758956 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: This master's current role is: LEADER
I20250904 20:00:40.759254 9112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: This master's current role is: LEADER
I20250904 20:00:40.776077 9063 tablet_replica.cc:331] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: stopping tablet replica
I20250904 20:00:40.776767 9063 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 LEADER]: Raft consensus shutting down.
I20250904 20:00:40.777299 9063 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 20:00:40.779973 9063 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250904 20:00:40.780520 9063 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250904 20:00:40.806478 9063 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250904 20:00:41.837081 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 8588
I20250904 20:00:41.871320 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 8721
I20250904 20:00:41.911160 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 8854
I20250904 20:00:41.942306 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38749
--webserver_interface=127.2.182.62
--webserver_port=38585
--builtin_ntp_servers=127.2.182.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.182.62:38749 with env {}
W20250904 20:00:42.261762 9121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:42.262424 9121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:42.262961 9121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:42.295428 9121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250904 20:00:42.295746 9121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:42.296016 9121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250904 20:00:42.296262 9121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250904 20:00:42.333309 9121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.182.62:38749
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.182.62:38749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.182.62
--webserver_port=38585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:42.334733 9121 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:42.336517 9121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:42.348626 9127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:43.750962 9126 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 9121
W20250904 20:00:42.349273 9128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:43.931535 9126 kernel_stack_watchdog.cc:198] Thread 9121 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250904 20:00:43.936372 9121 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.588s user 0.607s sys 0.981s
W20250904 20:00:43.936723 9129 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1582 milliseconds
W20250904 20:00:43.936820 9121 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.589s user 0.607s sys 0.981s
W20250904 20:00:43.938813 9130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:43.938895 9121 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:43.940140 9121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:43.942670 9121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:43.944118 9121 hybrid_clock.cc:648] HybridClock initialized: now 1757016043944071 us; error 59 us; skew 500 ppm
I20250904 20:00:43.944983 9121 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:43.951488 9121 webserver.cc:480] Webserver started at http://127.2.182.62:38585/ using document root <none> and password file <none>
I20250904 20:00:43.952548 9121 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:43.952798 9121 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:43.963395 9121 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20250904 20:00:43.968729 9137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:43.969967 9121 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250904 20:00:43.970371 9121 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
uuid: "307718c8edd342bb9dda5a72c84a6630"
format_stamp: "Formatted at 2025-09-04 20:00:40 on dist-test-slave-tjsx"
I20250904 20:00:43.972546 9121 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:44.031258 9121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:44.032793 9121 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:44.033262 9121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:44.109318 9121 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.62:38749
I20250904 20:00:44.109409 9188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.62:38749 every 8 connection(s)
I20250904 20:00:44.112371 9121 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/info.pb
I20250904 20:00:44.122367 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 9121
I20250904 20:00:44.123412 9189 sys_catalog.cc:263] Verifying existing consensus state
I20250904 20:00:44.123920 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.1:44495
--local_ip_for_outbound_sockets=127.2.182.1
--tserver_master_addrs=127.2.182.62:38749
--webserver_port=38989
--webserver_interface=127.2.182.1
--builtin_ntp_servers=127.2.182.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250904 20:00:44.136674 9189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Bootstrap starting.
I20250904 20:00:44.147284 9189 log.cc:826] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:44.160061 9189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 20:00:44.160872 9189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Bootstrap complete.
I20250904 20:00:44.182368 9189 raft_consensus.cc:357] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } }
I20250904 20:00:44.183152 9189 raft_consensus.cc:738] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307718c8edd342bb9dda5a72c84a6630, State: Initialized, Role: FOLLOWER
I20250904 20:00:44.183969 9189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } }
I20250904 20:00:44.184516 9189 raft_consensus.cc:397] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250904 20:00:44.184799 9189 raft_consensus.cc:491] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250904 20:00:44.185161 9189 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 1 FOLLOWER]: Advancing to term 2
I20250904 20:00:44.189481 9189 raft_consensus.cc:513] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } }
I20250904 20:00:44.190250 9189 leader_election.cc:304] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307718c8edd342bb9dda5a72c84a6630; no voters:
I20250904 20:00:44.192409 9189 leader_election.cc:290] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250904 20:00:44.192963 9193 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 20:00:44.196020 9193 raft_consensus.cc:695] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [term 2 LEADER]: Becoming Leader. State: Replica: 307718c8edd342bb9dda5a72c84a6630, State: Running, Role: LEADER
I20250904 20:00:44.197186 9189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: configured and running, proceeding with master startup.
I20250904 20:00:44.196914 9193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } }
I20250904 20:00:44.206758 9194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "307718c8edd342bb9dda5a72c84a6630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } } }
I20250904 20:00:44.207602 9194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: This master's current role is: LEADER
I20250904 20:00:44.209885 9195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307718c8edd342bb9dda5a72c84a6630. Latest consensus state: current_term: 2 leader_uuid: "307718c8edd342bb9dda5a72c84a6630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307718c8edd342bb9dda5a72c84a6630" member_type: VOTER last_known_addr { host: "127.2.182.62" port: 38749 } } }
I20250904 20:00:44.210584 9195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630 [sys.catalog]: This master's current role is: LEADER
I20250904 20:00:44.219190 9200 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250904 20:00:44.231564 9200 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=df79adbc5adc404595f31d0f4f2774f7]
I20250904 20:00:44.238885 9200 tablet_loader.cc:96] loaded metadata for tablet ccb113d82f82495fb34b46b81acb8337 (table pre_rebuild [id=df79adbc5adc404595f31d0f4f2774f7])
I20250904 20:00:44.240396 9200 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250904 20:00:44.261333 9200 catalog_manager.cc:1349] Generated new cluster ID: bb3f2ba9a727466aa3ea1ec0230c9eef
I20250904 20:00:44.261639 9200 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250904 20:00:44.300485 9200 catalog_manager.cc:1372] Generated new certificate authority record
I20250904 20:00:44.302023 9200 catalog_manager.cc:1506] Loading token signing keys...
I20250904 20:00:44.321334 9200 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Generated new TSK 0
I20250904 20:00:44.322420 9200 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250904 20:00:44.494076 9191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:44.494621 9191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:44.495193 9191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:44.530009 9191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:44.531059 9191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.1
I20250904 20:00:44.571126 9191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.1:44495
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.182.1
--webserver_port=38989
--tserver_master_addrs=127.2.182.62:38749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.1
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:44.572829 9191 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:44.574807 9191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:44.589756 9217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:45.991204 9216 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 9191
W20250904 20:00:46.091377 9191 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.501s user 0.605s sys 0.893s
W20250904 20:00:46.091840 9191 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.502s user 0.605s sys 0.893s
W20250904 20:00:44.593642 9218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:46.092468 9219 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250904 20:00:46.093583 9220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:46.093673 9191 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:46.097285 9191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:46.099776 9191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:46.101248 9191 hybrid_clock.cc:648] HybridClock initialized: now 1757016046101200 us; error 74 us; skew 500 ppm
I20250904 20:00:46.102133 9191 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:46.109531 9191 webserver.cc:480] Webserver started at http://127.2.182.1:38989/ using document root <none> and password file <none>
I20250904 20:00:46.110980 9191 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:46.111317 9191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:46.122197 9191 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20250904 20:00:46.127362 9228 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:46.128954 9191 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250904 20:00:46.129463 9191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
uuid: "f1d4552e31a64a6eb400751dd89f5f79"
format_stamp: "Formatted at 2025-09-04 20:00:28 on dist-test-slave-tjsx"
I20250904 20:00:46.131922 9191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:46.199491 9191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:46.201067 9191 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:46.201519 9191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:46.204224 9191 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:46.210497 9235 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 20:00:46.219678 9191 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 20:00:46.219990 9191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.000s
I20250904 20:00:46.220273 9191 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 20:00:46.225762 9191 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 20:00:46.226009 9191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.003s sys 0.000s
I20250904 20:00:46.226771 9235 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Bootstrap starting.
I20250904 20:00:46.399632 9191 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.1:44495
I20250904 20:00:46.399891 9341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.1:44495 every 8 connection(s)
I20250904 20:00:46.402500 9191 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/info.pb
I20250904 20:00:46.410415 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 9191
I20250904 20:00:46.412670 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.2:32953
--local_ip_for_outbound_sockets=127.2.182.2
--tserver_master_addrs=127.2.182.62:38749
--webserver_port=44355
--webserver_interface=127.2.182.2
--builtin_ntp_servers=127.2.182.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250904 20:00:46.433244 9342 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38749
I20250904 20:00:46.433717 9342 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:46.434800 9342 heartbeater.cc:507] Master 127.2.182.62:38749 requested a full tablet report, sending...
I20250904 20:00:46.439088 9154 ts_manager.cc:194] Registered new tserver with Master: f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1:44495)
I20250904 20:00:46.444497 9154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.1:41509
I20250904 20:00:46.484863 9235 log.cc:826] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Log is configured to *not* fsync() on all Append() calls
W20250904 20:00:46.768193 9346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:46.768795 9346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:46.769327 9346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:46.803818 9346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:46.804786 9346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.2
I20250904 20:00:46.841677 9346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.2:32953
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.182.2
--webserver_port=44355
--tserver_master_addrs=127.2.182.62:38749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.2
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:46.843235 9346 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:46.845067 9346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:46.858824 9353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:47.448683 9342 heartbeater.cc:499] Master 127.2.182.62:38749 was elected leader, sending a full tablet report...
W20250904 20:00:46.865378 9356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:46.861603 9354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:48.196185 9355 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1329 milliseconds
I20250904 20:00:48.196338 9346 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:48.197750 9346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:48.200959 9346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:48.202514 9346 hybrid_clock.cc:648] HybridClock initialized: now 1757016048202445 us; error 98 us; skew 500 ppm
I20250904 20:00:48.203578 9346 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:48.211925 9346 webserver.cc:480] Webserver started at http://127.2.182.2:44355/ using document root <none> and password file <none>
I20250904 20:00:48.213182 9346 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:48.213449 9346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:48.223495 9346 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20250904 20:00:48.229374 9363 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:48.230671 9346 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250904 20:00:48.231086 9346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
uuid: "4184c1efc269478880db76e7429daf9b"
format_stamp: "Formatted at 2025-09-04 20:00:30 on dist-test-slave-tjsx"
I20250904 20:00:48.233301 9346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:48.285722 9346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:48.287359 9346 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:48.287848 9346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:48.290824 9346 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:48.297386 9370 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 20:00:48.306125 9346 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 20:00:48.306381 9346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.000s
I20250904 20:00:48.306671 9346 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 20:00:48.311833 9346 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 20:00:48.312181 9346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.003s sys 0.003s
I20250904 20:00:48.312709 9370 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Bootstrap starting.
I20250904 20:00:48.527078 9346 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.2:32953
I20250904 20:00:48.527280 9476 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.2:32953 every 8 connection(s)
I20250904 20:00:48.531366 9346 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/info.pb
I20250904 20:00:48.539458 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 9346
I20250904 20:00:48.541556 2776 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
/tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.182.3:39333
--local_ip_for_outbound_sockets=127.2.182.3
--tserver_master_addrs=127.2.182.62:38749
--webserver_port=45853
--webserver_interface=127.2.182.3
--builtin_ntp_servers=127.2.182.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250904 20:00:48.579144 9477 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38749
I20250904 20:00:48.579735 9477 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:48.581260 9477 heartbeater.cc:507] Master 127.2.182.62:38749 requested a full tablet report, sending...
I20250904 20:00:48.586086 9154 ts_manager.cc:194] Registered new tserver with Master: 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953)
I20250904 20:00:48.589252 9154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.2:50443
I20250904 20:00:48.639986 9370 log.cc:826] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Log is configured to *not* fsync() on all Append() calls
W20250904 20:00:48.892988 9481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250904 20:00:48.893538 9481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250904 20:00:48.894052 9481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250904 20:00:48.928064 9481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250904 20:00:48.928988 9481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.182.3
I20250904 20:00:48.967491 9481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.182.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.182.3:39333
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.182.3
--webserver_port=45853
--tserver_master_addrs=127.2.182.62:38749
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.182.3
--log_dir=/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision db4f8e291cd478c794d4fd4cfb2d6832d781f88f
build type FASTDEBUG
built by None at 04 Sep 2025 19:43:21 UTC on 5fd53c4cbb9d
build id 7901
TSAN enabled
I20250904 20:00:48.967980 9235 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 20:00:48.969031 9481 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250904 20:00:48.969038 9235 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Bootstrap complete.
I20250904 20:00:48.970789 9235 ts_tablet_manager.cc:1397] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Time spent bootstrapping tablet: real 2.745s user 2.658s sys 0.084s
I20250904 20:00:48.970911 9481 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250904 20:00:48.991607 9489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250904 20:00:48.989580 9235 raft_consensus.cc:357] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:48.995209 9235 raft_consensus.cc:738] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d4552e31a64a6eb400751dd89f5f79, State: Initialized, Role: FOLLOWER
I20250904 20:00:48.996438 9235 consensus_queue.cc:260] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:49.013962 9235 ts_tablet_manager.cc:1428] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Time spent starting tablet: real 0.043s user 0.030s sys 0.008s
I20250904 20:00:49.594398 9477 heartbeater.cc:499] Master 127.2.182.62:38749 was elected leader, sending a full tablet report...
I20250904 20:00:50.228475 9495 raft_consensus.cc:491] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 20:00:50.229441 9495 raft_consensus.cc:513] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:50.252110 9495 leader_election.cc:290] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953), 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333)
W20250904 20:00:50.264413 9231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.182.3:39333: connect: Connection refused (error 111)
W20250904 20:00:50.280324 9231 leader_election.cc:336] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333): Network error: Client connection negotiation failed: client connection to 127.2.182.3:39333: connect: Connection refused (error 111)
I20250904 20:00:50.279753 9432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "f1d4552e31a64a6eb400751dd89f5f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "4184c1efc269478880db76e7429daf9b" is_pre_election: true
W20250904 20:00:50.293524 9230 leader_election.cc:343] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953): Illegal state: must be running to vote when last-logged opid is not known
I20250904 20:00:50.294028 9230 leader_election.cc:304] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1d4552e31a64a6eb400751dd89f5f79; no voters: 4184c1efc269478880db76e7429daf9b, 83c54ebbf0154f888587972d9b3187b7
I20250904 20:00:50.295552 9495 raft_consensus.cc:2747] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250904 20:00:48.993274 9488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:49.003540 9491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250904 20:00:50.331650 9490 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1340 milliseconds
I20250904 20:00:50.331847 9481 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250904 20:00:50.333530 9481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250904 20:00:50.336403 9481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250904 20:00:50.337924 9481 hybrid_clock.cc:648] HybridClock initialized: now 1757016050337883 us; error 39 us; skew 500 ppm
I20250904 20:00:50.339138 9481 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250904 20:00:50.346932 9481 webserver.cc:480] Webserver started at http://127.2.182.3:45853/ using document root <none> and password file <none>
I20250904 20:00:50.347962 9481 fs_manager.cc:362] Metadata directory not provided
I20250904 20:00:50.348202 9481 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250904 20:00:50.356756 9481 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250904 20:00:50.361922 9504 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250904 20:00:50.363170 9481 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250904 20:00:50.363519 9481 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
uuid: "83c54ebbf0154f888587972d9b3187b7"
format_stamp: "Formatted at 2025-09-04 20:00:32 on dist-test-slave-tjsx"
I20250904 20:00:50.365557 9481 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250904 20:00:50.419857 9481 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250904 20:00:50.421406 9481 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250904 20:00:50.421859 9481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250904 20:00:50.424880 9481 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250904 20:00:50.431627 9511 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250904 20:00:50.439510 9481 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250904 20:00:50.439755 9481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250904 20:00:50.440029 9481 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250904 20:00:50.445082 9481 ts_tablet_manager.cc:610] Registered 1 tablets
I20250904 20:00:50.445372 9481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250904 20:00:50.445828 9511 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Bootstrap starting.
I20250904 20:00:50.672358 9481 rpc_server.cc:307] RPC server started. Bound to: 127.2.182.3:39333
I20250904 20:00:50.672542 9617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.2.182.3:39333 every 8 connection(s)
I20250904 20:00:50.675621 9481 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/info.pb
I20250904 20:00:50.678556 2776 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu as pid 9481
I20250904 20:00:50.729387 9618 heartbeater.cc:344] Connected to a master server at 127.2.182.62:38749
I20250904 20:00:50.730036 9618 heartbeater.cc:461] Registering TS with master...
I20250904 20:00:50.731537 9618 heartbeater.cc:507] Master 127.2.182.62:38749 requested a full tablet report, sending...
I20250904 20:00:50.736315 9154 ts_manager.cc:194] Registered new tserver with Master: 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333)
I20250904 20:00:50.739858 9154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.182.3:33357
I20250904 20:00:50.743125 2776 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250904 20:00:50.774327 9511 log.cc:826] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Log is configured to *not* fsync() on all Append() calls
I20250904 20:00:51.494141 9370 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 20:00:51.495030 9370 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Bootstrap complete.
I20250904 20:00:51.496274 9370 ts_tablet_manager.cc:1397] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Time spent bootstrapping tablet: real 3.184s user 3.067s sys 0.092s
I20250904 20:00:51.502017 9370 raft_consensus.cc:357] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:51.504205 9370 raft_consensus.cc:738] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4184c1efc269478880db76e7429daf9b, State: Initialized, Role: FOLLOWER
I20250904 20:00:51.505067 9370 consensus_queue.cc:260] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:51.508270 9370 ts_tablet_manager.cc:1428] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20250904 20:00:51.745959 9618 heartbeater.cc:499] Master 127.2.182.62:38749 was elected leader, sending a full tablet report...
W20250904 20:00:51.974993 9185 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0d80 after lost signal to thread 9122
W20250904 20:00:51.975558 9185 debug-util.cc:398] Leaking SignalData structure 0x7b0800087380 after lost signal to thread 9188
I20250904 20:00:52.087843 9631 raft_consensus.cc:491] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 20:00:52.088374 9631 raft_consensus.cc:513] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:52.090359 9631 leader_election.cc:290] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953), 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333)
I20250904 20:00:52.091629 9432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "f1d4552e31a64a6eb400751dd89f5f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "4184c1efc269478880db76e7429daf9b" is_pre_election: true
I20250904 20:00:52.092571 9432 raft_consensus.cc:2466] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d4552e31a64a6eb400751dd89f5f79 in term 1.
I20250904 20:00:52.096094 9230 leader_election.cc:304] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4184c1efc269478880db76e7429daf9b, f1d4552e31a64a6eb400751dd89f5f79; no voters:
I20250904 20:00:52.097026 9631 raft_consensus.cc:2802] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250904 20:00:52.097602 9631 raft_consensus.cc:491] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 20:00:52.098032 9631 raft_consensus.cc:3058] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Advancing to term 2
I20250904 20:00:52.111589 9631 raft_consensus.cc:513] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:52.114729 9432 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "f1d4552e31a64a6eb400751dd89f5f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "4184c1efc269478880db76e7429daf9b"
I20250904 20:00:52.115365 9432 raft_consensus.cc:3058] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 1 FOLLOWER]: Advancing to term 2
I20250904 20:00:52.125301 9432 raft_consensus.cc:2466] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d4552e31a64a6eb400751dd89f5f79 in term 2.
I20250904 20:00:52.127447 9631 leader_election.cc:290] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 election: Requested vote from peers 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953), 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333)
I20250904 20:00:52.129000 9230 leader_election.cc:304] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4184c1efc269478880db76e7429daf9b, f1d4552e31a64a6eb400751dd89f5f79; no voters:
I20250904 20:00:52.129956 9631 raft_consensus.cc:2802] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 2 FOLLOWER]: Leader election won for term 2
I20250904 20:00:52.123453 9573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "f1d4552e31a64a6eb400751dd89f5f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "83c54ebbf0154f888587972d9b3187b7" is_pre_election: true
I20250904 20:00:52.128123 9572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccb113d82f82495fb34b46b81acb8337" candidate_uuid: "f1d4552e31a64a6eb400751dd89f5f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "83c54ebbf0154f888587972d9b3187b7"
W20250904 20:00:52.138108 9231 leader_election.cc:343] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333): Illegal state: must be running to vote when last-logged opid is not known
I20250904 20:00:52.140442 9631 raft_consensus.cc:695] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 2 LEADER]: Becoming Leader. State: Replica: f1d4552e31a64a6eb400751dd89f5f79, State: Running, Role: LEADER
I20250904 20:00:52.141607 9631 consensus_queue.cc:237] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:52.156159 9153 catalog_manager.cc:5582] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 reported cstate change: term changed from 0 to 2, leader changed from <none> to f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1), VOTER 4184c1efc269478880db76e7429daf9b (127.2.182.2) added, VOTER 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3) added, VOTER f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1) added. New cstate: current_term: 2 leader_uuid: "f1d4552e31a64a6eb400751dd89f5f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } health_report { overall_health: HEALTHY } } }
W20250904 20:00:52.536998 9231 consensus_peers.cc:489] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 -> Peer 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333): Couldn't send request to peer 83c54ebbf0154f888587972d9b3187b7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250904 20:00:52.680688 9432 raft_consensus.cc:1273] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 2 FOLLOWER]: Refusing update from remote peer f1d4552e31a64a6eb400751dd89f5f79: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250904 20:00:52.682782 9631 consensus_queue.cc:1035] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250904 20:00:52.739694 2776 scanner-internal.cc:458] Time spent opening tablet: real 1.945s user 0.008s sys 0.004s
I20250904 20:00:52.823977 9297 consensus_queue.cc:237] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:52.830341 9432 raft_consensus.cc:1273] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 2 FOLLOWER]: Refusing update from remote peer f1d4552e31a64a6eb400751dd89f5f79: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250904 20:00:52.832176 9647 consensus_queue.cc:1035] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250904 20:00:52.839882 9647 raft_consensus.cc:2953] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } } }
I20250904 20:00:52.843196 9431 raft_consensus.cc:2953] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } } }
I20250904 20:00:52.856714 9141 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ccb113d82f82495fb34b46b81acb8337 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250904 20:00:52.859452 9153 catalog_manager.cc:5582] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b reported cstate change: config changed from index -1 to 207, VOTER 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3) evicted. New cstate: current_term: 2 leader_uuid: "f1d4552e31a64a6eb400751dd89f5f79" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } } }
I20250904 20:00:52.918627 9297 consensus_queue.cc:237] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:52.924845 9647 raft_consensus.cc:2953] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 4184c1efc269478880db76e7429daf9b (127.2.182.2) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } } }
I20250904 20:00:52.940101 9141 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ccb113d82f82495fb34b46b81acb8337 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250904 20:00:52.944861 9154 catalog_manager.cc:5582] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 reported cstate change: config changed from index 207 to 208, VOTER 4184c1efc269478880db76e7429daf9b (127.2.182.2) evicted. New cstate: current_term: 2 leader_uuid: "f1d4552e31a64a6eb400751dd89f5f79" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } health_report { overall_health: HEALTHY } } }
I20250904 20:00:52.947074 9553 tablet_service.cc:1515] Processing DeleteTablet for tablet ccb113d82f82495fb34b46b81acb8337 with delete_type TABLET_DATA_TOMBSTONED (TS 83c54ebbf0154f888587972d9b3187b7 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:58174
W20250904 20:00:52.973847 9140 catalog_manager.cc:4908] TS 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333): delete failed for tablet ccb113d82f82495fb34b46b81acb8337 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccb113d82f82495fb34b46b81acb8337 already in progress: opening tablet
I20250904 20:00:52.993669 9412 tablet_service.cc:1515] Processing DeleteTablet for tablet ccb113d82f82495fb34b46b81acb8337 with delete_type TABLET_DATA_TOMBSTONED (TS 4184c1efc269478880db76e7429daf9b not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:41324
I20250904 20:00:53.014421 9657 tablet_replica.cc:331] stopping tablet replica
I20250904 20:00:53.015367 9657 raft_consensus.cc:2241] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250904 20:00:53.016108 9657 raft_consensus.cc:2270] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250904 20:00:53.073606 9657 ts_tablet_manager.cc:1905] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250904 20:00:53.102593 9657 ts_tablet_manager.cc:1918] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250904 20:00:53.115190 9657 log.cc:1199] T ccb113d82f82495fb34b46b81acb8337 P 4184c1efc269478880db76e7429daf9b: Deleting WAL directory at /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/wal/wals/ccb113d82f82495fb34b46b81acb8337
I20250904 20:00:53.117326 9139 catalog_manager.cc:4928] TS 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953): tablet ccb113d82f82495fb34b46b81acb8337 (table pre_rebuild [id=df79adbc5adc404595f31d0f4f2774f7]) successfully deleted
I20250904 20:00:53.267386 9511 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250904 20:00:53.268462 9511 tablet_bootstrap.cc:492] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Bootstrap complete.
I20250904 20:00:53.270650 9511 ts_tablet_manager.cc:1397] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Time spent bootstrapping tablet: real 2.825s user 2.691s sys 0.068s
I20250904 20:00:53.278609 9511 raft_consensus.cc:357] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:53.281752 9511 raft_consensus.cc:738] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c54ebbf0154f888587972d9b3187b7, State: Initialized, Role: FOLLOWER
I20250904 20:00:53.283430 9511 consensus_queue.cc:260] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:53.294170 9511 ts_tablet_manager.cc:1428] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Time spent starting tablet: real 0.023s user 0.016s sys 0.004s
I20250904 20:00:53.300334 9553 tablet_service.cc:1515] Processing DeleteTablet for tablet ccb113d82f82495fb34b46b81acb8337 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:58174
I20250904 20:00:53.301754 9655 tablet_replica.cc:331] stopping tablet replica
I20250904 20:00:53.302801 9655 raft_consensus.cc:2241] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250904 20:00:53.303517 9655 raft_consensus.cc:2270] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250904 20:00:53.342550 9655 ts_tablet_manager.cc:1905] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250904 20:00:53.363853 9655 ts_tablet_manager.cc:1918] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250904 20:00:53.364344 9655 log.cc:1199] T ccb113d82f82495fb34b46b81acb8337 P 83c54ebbf0154f888587972d9b3187b7: Deleting WAL directory at /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/wal/wals/ccb113d82f82495fb34b46b81acb8337
I20250904 20:00:53.366235 9140 catalog_manager.cc:4928] TS 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3:39333): tablet ccb113d82f82495fb34b46b81acb8337 (table pre_rebuild [id=df79adbc5adc404595f31d0f4f2774f7]) successfully deleted
I20250904 20:00:53.867898 9553 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:00:53.894521 9412 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:00:53.899993 9277 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
307718c8edd342bb9dda5a72c84a6630 | 127.2.182.62:38749 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.182.20:33131 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
4184c1efc269478880db76e7429daf9b | 127.2.182.2:32953 | HEALTHY | <none> | 0 | 0
83c54ebbf0154f888587972d9b3187b7 | 127.2.182.3:39333 | HEALTHY | <none> | 0 | 0
f1d4552e31a64a6eb400751dd89f5f79 | 127.2.182.1:44495 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.182.1 | experimental | 127.2.182.1:44495
local_ip_for_outbound_sockets | 127.2.182.2 | experimental | 127.2.182.2:32953
local_ip_for_outbound_sockets | 127.2.182.3 | experimental | 127.2.182.3:39333
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/info.pb | hidden | 127.2.182.1:44495
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/info.pb | hidden | 127.2.182.2:32953
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/info.pb | hidden | 127.2.182.3:39333
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.182.20:33131 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250904 20:00:54.289975 2776 log_verifier.cc:126] Checking tablet ccb113d82f82495fb34b46b81acb8337
I20250904 20:00:54.650322 2776 log_verifier.cc:177] Verified matching terms for 208 ops in tablet ccb113d82f82495fb34b46b81acb8337
I20250904 20:00:54.653475 9153 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33770:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250904 20:00:54.654215 9153 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33770:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250904 20:00:54.670110 9153 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 307718c8edd342bb9dda5a72c84a6630: Sending DeleteTablet for 1 replicas of tablet ccb113d82f82495fb34b46b81acb8337
I20250904 20:00:54.672739 2776 test_util.cc:276] Using random seed: 768453950
I20250904 20:00:54.673758 9277 tablet_service.cc:1515] Processing DeleteTablet for tablet ccb113d82f82495fb34b46b81acb8337 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-04 20:00:54 UTC) from {username='slave'} at 127.0.0.1:36060
I20250904 20:00:54.691604 9691 tablet_replica.cc:331] stopping tablet replica
I20250904 20:00:54.692847 9691 raft_consensus.cc:2241] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 2 LEADER]: Raft consensus shutting down.
I20250904 20:00:54.694203 9691 raft_consensus.cc:2270] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250904 20:00:54.742396 9153 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40658:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250904 20:00:54.745772 9153 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250904 20:00:54.747953 9691 ts_tablet_manager.cc:1905] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250904 20:00:54.767908 9691 ts_tablet_manager.cc:1918] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250904 20:00:54.768656 9691 log.cc:1199] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Deleting WAL directory at /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/wal/wals/ccb113d82f82495fb34b46b81acb8337
I20250904 20:00:54.769855 9691 ts_tablet_manager.cc:1939] T ccb113d82f82495fb34b46b81acb8337 P f1d4552e31a64a6eb400751dd89f5f79: Deleting consensus metadata
I20250904 20:00:54.772764 9553 tablet_service.cc:1468] Processing CreateTablet for tablet a34e3899c34646c8ac286f39fd60fc70 (DEFAULT_TABLE table=post_rebuild [id=50f1d40b29b8479ebd41192e61774365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:54.774268 9553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34e3899c34646c8ac286f39fd60fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:54.774390 9277 tablet_service.cc:1468] Processing CreateTablet for tablet a34e3899c34646c8ac286f39fd60fc70 (DEFAULT_TABLE table=post_rebuild [id=50f1d40b29b8479ebd41192e61774365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:54.776180 9277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34e3899c34646c8ac286f39fd60fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:54.780735 9412 tablet_service.cc:1468] Processing CreateTablet for tablet a34e3899c34646c8ac286f39fd60fc70 (DEFAULT_TABLE table=post_rebuild [id=50f1d40b29b8479ebd41192e61774365]), partition=RANGE (key) PARTITION UNBOUNDED
I20250904 20:00:54.782392 9412 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34e3899c34646c8ac286f39fd60fc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250904 20:00:54.787736 9141 catalog_manager.cc:4928] TS f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1:44495): tablet ccb113d82f82495fb34b46b81acb8337 (table pre_rebuild [id=df79adbc5adc404595f31d0f4f2774f7]) successfully deleted
I20250904 20:00:54.806839 9699 tablet_bootstrap.cc:492] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7: Bootstrap starting.
I20250904 20:00:54.803982 9698 tablet_bootstrap.cc:492] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79: Bootstrap starting.
I20250904 20:00:54.810808 9700 tablet_bootstrap.cc:492] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b: Bootstrap starting.
I20250904 20:00:54.813558 9699 tablet_bootstrap.cc:654] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:54.819351 9698 tablet_bootstrap.cc:654] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:54.820815 9700 tablet_bootstrap.cc:654] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b: Neither blocks nor log segments found. Creating new log.
I20250904 20:00:54.825517 9699 tablet_bootstrap.cc:492] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7: No bootstrap required, opened a new log
I20250904 20:00:54.826064 9699 ts_tablet_manager.cc:1397] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20250904 20:00:54.829124 9699 raft_consensus.cc:357] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.830075 9699 raft_consensus.cc:383] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:54.830431 9699 raft_consensus.cc:738] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c54ebbf0154f888587972d9b3187b7, State: Initialized, Role: FOLLOWER
I20250904 20:00:54.831895 9699 consensus_queue.cc:260] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.836611 9698 tablet_bootstrap.cc:492] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79: No bootstrap required, opened a new log
I20250904 20:00:54.837236 9698 ts_tablet_manager.cc:1397] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.004s
I20250904 20:00:54.840780 9698 raft_consensus.cc:357] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.842082 9700 tablet_bootstrap.cc:492] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b: No bootstrap required, opened a new log
I20250904 20:00:54.842005 9698 raft_consensus.cc:383] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:54.842574 9700 ts_tablet_manager.cc:1397] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.007s
I20250904 20:00:54.842448 9698 raft_consensus.cc:738] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d4552e31a64a6eb400751dd89f5f79, State: Initialized, Role: FOLLOWER
I20250904 20:00:54.843396 9698 consensus_queue.cc:260] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.846263 9699 ts_tablet_manager.cc:1428] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7: Time spent starting tablet: real 0.020s user 0.006s sys 0.004s
I20250904 20:00:54.849426 9704 raft_consensus.cc:491] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250904 20:00:54.850072 9704 raft_consensus.cc:513] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.845595 9700 raft_consensus.cc:357] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.855077 9700 raft_consensus.cc:383] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250904 20:00:54.855469 9700 raft_consensus.cc:738] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4184c1efc269478880db76e7429daf9b, State: Initialized, Role: FOLLOWER
I20250904 20:00:54.855868 9704 leader_election.cc:290] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953), f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1:44495)
I20250904 20:00:54.856676 9700 consensus_queue.cc:260] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.860539 9700 ts_tablet_manager.cc:1428] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b: Time spent starting tablet: real 0.018s user 0.004s sys 0.003s
I20250904 20:00:54.870162 9698 ts_tablet_manager.cc:1428] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79: Time spent starting tablet: real 0.032s user 0.000s sys 0.014s
I20250904 20:00:54.879779 9431 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34e3899c34646c8ac286f39fd60fc70" candidate_uuid: "83c54ebbf0154f888587972d9b3187b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4184c1efc269478880db76e7429daf9b" is_pre_election: true
I20250904 20:00:54.880494 9431 raft_consensus.cc:2466] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c54ebbf0154f888587972d9b3187b7 in term 0.
I20250904 20:00:54.881891 9506 leader_election.cc:304] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4184c1efc269478880db76e7429daf9b, 83c54ebbf0154f888587972d9b3187b7; no voters:
I20250904 20:00:54.882673 9704 raft_consensus.cc:2802] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250904 20:00:54.882385 9297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34e3899c34646c8ac286f39fd60fc70" candidate_uuid: "83c54ebbf0154f888587972d9b3187b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d4552e31a64a6eb400751dd89f5f79" is_pre_election: true
I20250904 20:00:54.883019 9704 raft_consensus.cc:491] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250904 20:00:54.883328 9704 raft_consensus.cc:3058] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:54.883343 9297 raft_consensus.cc:2466] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c54ebbf0154f888587972d9b3187b7 in term 0.
I20250904 20:00:54.889320 9704 raft_consensus.cc:513] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.890899 9704 leader_election.cc:290] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [CANDIDATE]: Term 1 election: Requested vote from peers 4184c1efc269478880db76e7429daf9b (127.2.182.2:32953), f1d4552e31a64a6eb400751dd89f5f79 (127.2.182.1:44495)
I20250904 20:00:54.891808 9431 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34e3899c34646c8ac286f39fd60fc70" candidate_uuid: "83c54ebbf0154f888587972d9b3187b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4184c1efc269478880db76e7429daf9b"
I20250904 20:00:54.892055 9297 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34e3899c34646c8ac286f39fd60fc70" candidate_uuid: "83c54ebbf0154f888587972d9b3187b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1d4552e31a64a6eb400751dd89f5f79"
I20250904 20:00:54.892400 9431 raft_consensus.cc:3058] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:54.892585 9297 raft_consensus.cc:3058] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79 [term 0 FOLLOWER]: Advancing to term 1
I20250904 20:00:54.899679 9297 raft_consensus.cc:2466] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c54ebbf0154f888587972d9b3187b7 in term 1.
I20250904 20:00:54.899670 9431 raft_consensus.cc:2466] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c54ebbf0154f888587972d9b3187b7 in term 1.
I20250904 20:00:54.900864 9506 leader_election.cc:304] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4184c1efc269478880db76e7429daf9b, 83c54ebbf0154f888587972d9b3187b7; no voters:
I20250904 20:00:54.901625 9704 raft_consensus.cc:2802] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 1 FOLLOWER]: Leader election won for term 1
I20250904 20:00:54.903508 9704 raft_consensus.cc:695] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [term 1 LEADER]: Becoming Leader. State: Replica: 83c54ebbf0154f888587972d9b3187b7, State: Running, Role: LEADER
I20250904 20:00:54.904486 9704 consensus_queue.cc:237] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } }
I20250904 20:00:54.918664 9153 catalog_manager.cc:5582] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c54ebbf0154f888587972d9b3187b7 (127.2.182.3). New cstate: current_term: 1 leader_uuid: "83c54ebbf0154f888587972d9b3187b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c54ebbf0154f888587972d9b3187b7" member_type: VOTER last_known_addr { host: "127.2.182.3" port: 39333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 } health_report { overall_health: UNKNOWN } } }
W20250904 20:00:54.932576 9343 tablet.cc:2378] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 20:00:54.953964 9619 tablet.cc:2378] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250904 20:00:55.049315 9478 tablet.cc:2378] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250904 20:00:55.109946 9431 raft_consensus.cc:1273] T a34e3899c34646c8ac286f39fd60fc70 P 4184c1efc269478880db76e7429daf9b [term 1 FOLLOWER]: Refusing update from remote peer 83c54ebbf0154f888587972d9b3187b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 20:00:55.109922 9297 raft_consensus.cc:1273] T a34e3899c34646c8ac286f39fd60fc70 P f1d4552e31a64a6eb400751dd89f5f79 [term 1 FOLLOWER]: Refusing update from remote peer 83c54ebbf0154f888587972d9b3187b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250904 20:00:55.111979 9704 consensus_queue.cc:1035] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4184c1efc269478880db76e7429daf9b" member_type: VOTER last_known_addr { host: "127.2.182.2" port: 32953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250904 20:00:55.112990 9711 consensus_queue.cc:1035] T a34e3899c34646c8ac286f39fd60fc70 P 83c54ebbf0154f888587972d9b3187b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1d4552e31a64a6eb400751dd89f5f79" member_type: VOTER last_known_addr { host: "127.2.182.1" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250904 20:00:55.145059 9719 mvcc.cc:204] Tried to move back new op lower bound from 7196737761712705536 to 7196737760899358720. Current Snapshot: MvccSnapshot[applied={T|T < 7196737761712705536}]
I20250904 20:00:55.179533 9720 mvcc.cc:204] Tried to move back new op lower bound from 7196737761712705536 to 7196737760899358720. Current Snapshot: MvccSnapshot[applied={T|T < 7196737761712705536}]
I20250904 20:01:00.582976 9277 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:01:00.583900 9412 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250904 20:01:00.584713 9553 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
307718c8edd342bb9dda5a72c84a6630 | 127.2.182.62:38749 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.182.20:33131 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
4184c1efc269478880db76e7429daf9b | 127.2.182.2:32953 | HEALTHY | <none> | 0 | 0
83c54ebbf0154f888587972d9b3187b7 | 127.2.182.3:39333 | HEALTHY | <none> | 1 | 0
f1d4552e31a64a6eb400751dd89f5f79 | 127.2.182.1:44495 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.2.182.1 | experimental | 127.2.182.1:44495
local_ip_for_outbound_sockets | 127.2.182.2 | experimental | 127.2.182.2:32953
local_ip_for_outbound_sockets | 127.2.182.3 | experimental | 127.2.182.3:39333
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-0/data/info.pb | hidden | 127.2.182.1:44495
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-1/data/info.pb | hidden | 127.2.182.2:32953
server_dump_info_path | /tmp/dist-test-taskjPH3LF/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757015875386078-2776-0/minicluster-data/ts-2/data/info.pb | hidden | 127.2.182.3:39333
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.2.182.20:33131 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250904 20:01:00.848708 2776 log_verifier.cc:126] Checking tablet a34e3899c34646c8ac286f39fd60fc70
I20250904 20:01:01.625054 2776 log_verifier.cc:177] Verified matching terms for 205 ops in tablet a34e3899c34646c8ac286f39fd60fc70
I20250904 20:01:01.626103 2776 log_verifier.cc:126] Checking tablet ccb113d82f82495fb34b46b81acb8337
I20250904 20:01:01.626426 2776 log_verifier.cc:177] Verified matching terms for 0 ops in tablet ccb113d82f82495fb34b46b81acb8337
I20250904 20:01:01.652210 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 9191
I20250904 20:01:01.691723 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 9346
I20250904 20:01:01.724874 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 9481
I20250904 20:01:01.764986 2776 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskjPH3LF/build/tsan/bin/kudu with pid 9121
2025-09-04T20:01:01Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (37976 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (37976 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (186381 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST