Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250812 13:57:46.796984 4728 test_util.cc:276] Using random seed: 1825272636
W20250812 13:57:47.917378 4728 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.079s user 0.498s sys 0.578s
W20250812 13:57:47.917719 4728 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.080s user 0.498s sys 0.578s
I20250812 13:57:47.922835 4728 ts_itest-base.cc:115] Starting cluster with:
I20250812 13:57:47.923010 4728 ts_itest-base.cc:116] --------------
I20250812 13:57:47.923164 4728 ts_itest-base.cc:117] 4 tablet servers
I20250812 13:57:47.923321 4728 ts_itest-base.cc:118] 3 replicas per TS
I20250812 13:57:47.923458 4728 ts_itest-base.cc:119] --------------
2025-08-12T13:57:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T13:57:47Z Disabled control of system clock
I20250812 13:57:47.956386 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:46075
--webserver_interface=127.4.158.62
--webserver_port=0
--builtin_ntp_servers=127.4.158.20:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:46075 with env {}
W20250812 13:57:48.243135 4742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:57:48.243716 4742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:57:48.244153 4742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:57:48.274832 4742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:57:48.275137 4742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:57:48.275388 4742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:57:48.275621 4742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:57:48.310226 4742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:46075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:57:48.311612 4742 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:57:48.313218 4742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:57:48.323719 4748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:48.324000 4749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:57:48.328094 4742 server_base.cc:1047] running on GCE node
W20250812 13:57:48.327908 4751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:57:49.409780 4742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:57:49.412443 4742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:57:49.413774 4742 hybrid_clock.cc:648] HybridClock initialized: now 1755007069413747 us; error 45 us; skew 500 ppm
I20250812 13:57:49.414592 4742 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:57:49.420480 4742 webserver.cc:480] Webserver started at http://127.4.158.62:34381/ using document root <none> and password file <none>
I20250812 13:57:49.421514 4742 fs_manager.cc:362] Metadata directory not provided
I20250812 13:57:49.421746 4742 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:57:49.422240 4742 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:57:49.426637 4742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "36fb135f8c044dd48cf762a42bc2ff74"
format_stamp: "Formatted at 2025-08-12 13:57:49 on dist-test-slave-f6np"
I20250812 13:57:49.427670 4742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "36fb135f8c044dd48cf762a42bc2ff74"
format_stamp: "Formatted at 2025-08-12 13:57:49 on dist-test-slave-f6np"
I20250812 13:57:49.434620 4742 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250812 13:57:49.440733 4758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:49.441721 4742 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250812 13:57:49.442058 4742 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "36fb135f8c044dd48cf762a42bc2ff74"
format_stamp: "Formatted at 2025-08-12 13:57:49 on dist-test-slave-f6np"
I20250812 13:57:49.442394 4742 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:57:49.491951 4742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:57:49.493408 4742 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:57:49.493829 4742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:57:49.560854 4742 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:46075
I20250812 13:57:49.560918 4809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:46075 every 8 connection(s)
I20250812 13:57:49.563511 4742 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:57:49.568426 4810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:57:49.568594 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 4742
I20250812 13:57:49.568974 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 13:57:49.586289 4810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Bootstrap starting.
I20250812 13:57:49.591961 4810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Neither blocks nor log segments found. Creating new log.
I20250812 13:57:49.594295 4810 log.cc:826] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Log is configured to *not* fsync() on all Append() calls
I20250812 13:57:49.599167 4810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: No bootstrap required, opened a new log
I20250812 13:57:49.622028 4810 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } }
I20250812 13:57:49.622831 4810 raft_consensus.cc:383] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:57:49.623101 4810 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fb135f8c044dd48cf762a42bc2ff74, State: Initialized, Role: FOLLOWER
I20250812 13:57:49.623859 4810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } }
I20250812 13:57:49.624359 4810 raft_consensus.cc:397] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:57:49.624614 4810 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:57:49.624905 4810 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:57:49.628852 4810 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } }
I20250812 13:57:49.629500 4810 leader_election.cc:304] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36fb135f8c044dd48cf762a42bc2ff74; no voters:
I20250812 13:57:49.631392 4810 leader_election.cc:290] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:57:49.632149 4815 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:57:49.634099 4815 raft_consensus.cc:695] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 1 LEADER]: Becoming Leader. State: Replica: 36fb135f8c044dd48cf762a42bc2ff74, State: Running, Role: LEADER
I20250812 13:57:49.634904 4815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } }
I20250812 13:57:49.635324 4810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:57:49.644750 4817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36fb135f8c044dd48cf762a42bc2ff74. Latest consensus state: current_term: 1 leader_uuid: "36fb135f8c044dd48cf762a42bc2ff74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } } }
I20250812 13:57:49.644951 4816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36fb135f8c044dd48cf762a42bc2ff74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } } }
I20250812 13:57:49.645471 4817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: This master's current role is: LEADER
I20250812 13:57:49.645556 4816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: This master's current role is: LEADER
I20250812 13:57:49.649457 4824 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:57:49.659737 4824 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:57:49.675473 4824 catalog_manager.cc:1349] Generated new cluster ID: 7501bfc9943d477aa85de342d29f7225
I20250812 13:57:49.675733 4824 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:57:49.703539 4824 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 13:57:49.704962 4824 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:57:49.724947 4824 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Generated new TSK 0
I20250812 13:57:49.726109 4824 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 13:57:49.744691 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:0
--local_ip_for_outbound_sockets=127.4.158.1
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46075
--builtin_ntp_servers=127.4.158.20:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 13:57:50.040489 4834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:57:50.040985 4834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:57:50.041476 4834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:57:50.072257 4834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:57:50.073165 4834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 13:57:50.107681 4834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:57:50.109151 4834 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:57:50.110738 4834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:57:50.123073 4840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:51.525092 4839 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4834
W20250812 13:57:51.572504 4834 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.449s user 0.513s sys 0.920s
W20250812 13:57:51.572253 4842 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1447 milliseconds
W20250812 13:57:51.573729 4834 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.450s user 0.514s sys 0.921s
W20250812 13:57:50.124699 4841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:57:51.574265 4834 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250812 13:57:51.574362 4843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:57:51.578595 4834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:57:51.581004 4834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:57:51.582404 4834 hybrid_clock.cc:648] HybridClock initialized: now 1755007071582360 us; error 55 us; skew 500 ppm
I20250812 13:57:51.583168 4834 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:57:51.589375 4834 webserver.cc:480] Webserver started at http://127.4.158.1:38649/ using document root <none> and password file <none>
I20250812 13:57:51.590305 4834 fs_manager.cc:362] Metadata directory not provided
I20250812 13:57:51.590502 4834 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:57:51.590941 4834 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:57:51.595535 4834 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ab9c948b6f0e4ba98971df43318d7228"
format_stamp: "Formatted at 2025-08-12 13:57:51 on dist-test-slave-f6np"
I20250812 13:57:51.596627 4834 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ab9c948b6f0e4ba98971df43318d7228"
format_stamp: "Formatted at 2025-08-12 13:57:51 on dist-test-slave-f6np"
I20250812 13:57:51.604041 4834 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.005s
I20250812 13:57:51.610587 4850 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:51.612679 4834 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20250812 13:57:51.613082 4834 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ab9c948b6f0e4ba98971df43318d7228"
format_stamp: "Formatted at 2025-08-12 13:57:51 on dist-test-slave-f6np"
I20250812 13:57:51.613467 4834 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:57:51.663537 4834 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:57:51.665012 4834 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:57:51.665441 4834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:57:51.668326 4834 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:57:51.672487 4834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:57:51.672696 4834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:51.672924 4834 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:57:51.673077 4834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:51.826393 4834 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:35893
I20250812 13:57:51.826500 4962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:35893 every 8 connection(s)
I20250812 13:57:51.829000 4834 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 13:57:51.830569 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 4834
I20250812 13:57:51.831074 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 13:57:51.839448 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:0
--local_ip_for_outbound_sockets=127.4.158.2
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46075
--builtin_ntp_servers=127.4.158.20:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:57:51.872897 4963 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46075
I20250812 13:57:51.873371 4963 heartbeater.cc:461] Registering TS with master...
I20250812 13:57:51.874493 4963 heartbeater.cc:507] Master 127.4.158.62:46075 requested a full tablet report, sending...
I20250812 13:57:51.876926 4775 ts_manager.cc:194] Registered new tserver with Master: ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1:35893)
I20250812 13:57:51.879235 4775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:40001
W20250812 13:57:52.154898 4967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:57:52.155398 4967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:57:52.155890 4967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:57:52.187256 4967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:57:52.188092 4967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 13:57:52.223140 4967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:57:52.224476 4967 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:57:52.226099 4967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:57:52.238324 4973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:57:52.882586 4963 heartbeater.cc:499] Master 127.4.158.62:46075 was elected leader, sending a full tablet report...
W20250812 13:57:52.239270 4974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:53.343514 4976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:53.345829 4975 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1102 milliseconds
I20250812 13:57:53.345937 4967 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:57:53.347076 4967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:57:53.349534 4967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:57:53.350946 4967 hybrid_clock.cc:648] HybridClock initialized: now 1755007073350889 us; error 51 us; skew 500 ppm
I20250812 13:57:53.351747 4967 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:57:53.357570 4967 webserver.cc:480] Webserver started at http://127.4.158.2:38801/ using document root <none> and password file <none>
I20250812 13:57:53.358505 4967 fs_manager.cc:362] Metadata directory not provided
I20250812 13:57:53.358696 4967 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:57:53.359148 4967 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:57:53.363430 4967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8deec1959abc41ee89fdab731a18ede3"
format_stamp: "Formatted at 2025-08-12 13:57:53 on dist-test-slave-f6np"
I20250812 13:57:53.364495 4967 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8deec1959abc41ee89fdab731a18ede3"
format_stamp: "Formatted at 2025-08-12 13:57:53 on dist-test-slave-f6np"
I20250812 13:57:53.371307 4967 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250812 13:57:53.376724 4983 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:53.377753 4967 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250812 13:57:53.378093 4967 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8deec1959abc41ee89fdab731a18ede3"
format_stamp: "Formatted at 2025-08-12 13:57:53 on dist-test-slave-f6np"
I20250812 13:57:53.378400 4967 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:57:53.435732 4967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:57:53.437278 4967 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:57:53.437702 4967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:57:53.440176 4967 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:57:53.444133 4967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:57:53.444347 4967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:53.444589 4967 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:57:53.444744 4967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:53.575945 4967 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:44397
I20250812 13:57:53.576043 5095 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:44397 every 8 connection(s)
I20250812 13:57:53.578547 4967 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 13:57:53.581254 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 4967
I20250812 13:57:53.581753 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 13:57:53.588083 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:0
--local_ip_for_outbound_sockets=127.4.158.3
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46075
--builtin_ntp_servers=127.4.158.20:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:57:53.599061 5096 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46075
I20250812 13:57:53.599493 5096 heartbeater.cc:461] Registering TS with master...
I20250812 13:57:53.600569 5096 heartbeater.cc:507] Master 127.4.158.62:46075 requested a full tablet report, sending...
I20250812 13:57:53.602846 4775 ts_manager.cc:194] Registered new tserver with Master: 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397)
I20250812 13:57:53.604526 4775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:60677
W20250812 13:57:53.876128 5100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:57:53.876588 5100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:57:53.877015 5100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:57:53.908576 5100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:57:53.909363 5100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 13:57:53.943159 5100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:57:53.944367 5100 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:57:53.945879 5100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:57:53.956856 5106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:57:54.607746 5096 heartbeater.cc:499] Master 127.4.158.62:46075 was elected leader, sending a full tablet report...
W20250812 13:57:53.958240 5107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:55.037536 5109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:55.039685 5108 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1077 milliseconds
I20250812 13:57:55.039775 5100 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:57:55.040918 5100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:57:55.043439 5100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:57:55.044835 5100 hybrid_clock.cc:648] HybridClock initialized: now 1755007075044806 us; error 36 us; skew 500 ppm
I20250812 13:57:55.045616 5100 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:57:55.053757 5100 webserver.cc:480] Webserver started at http://127.4.158.3:32877/ using document root <none> and password file <none>
I20250812 13:57:55.054670 5100 fs_manager.cc:362] Metadata directory not provided
I20250812 13:57:55.054872 5100 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:57:55.055346 5100 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:57:55.060410 5100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6e87defbedd749738d8fd9485c9c345f"
format_stamp: "Formatted at 2025-08-12 13:57:55 on dist-test-slave-f6np"
I20250812 13:57:55.062129 5100 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6e87defbedd749738d8fd9485c9c345f"
format_stamp: "Formatted at 2025-08-12 13:57:55 on dist-test-slave-f6np"
I20250812 13:57:55.070484 5100 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20250812 13:57:55.077584 5116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:55.078693 5100 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250812 13:57:55.079039 5100 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6e87defbedd749738d8fd9485c9c345f"
format_stamp: "Formatted at 2025-08-12 13:57:55 on dist-test-slave-f6np"
I20250812 13:57:55.079453 5100 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:57:55.149672 5100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:57:55.151171 5100 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:57:55.151599 5100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:57:55.154156 5100 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:57:55.158020 5100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:57:55.158238 5100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:55.158459 5100 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:57:55.158608 5100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:55.280359 5100 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:38573
I20250812 13:57:55.280455 5228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:38573 every 8 connection(s)
I20250812 13:57:55.282836 5100 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 13:57:55.284193 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 5100
I20250812 13:57:55.284576 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 13:57:55.291707 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.4:0
--local_ip_for_outbound_sockets=127.4.158.4
--webserver_interface=127.4.158.4
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46075
--builtin_ntp_servers=127.4.158.20:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:57:55.305032 5229 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46075
I20250812 13:57:55.305426 5229 heartbeater.cc:461] Registering TS with master...
I20250812 13:57:55.306421 5229 heartbeater.cc:507] Master 127.4.158.62:46075 requested a full tablet report, sending...
I20250812 13:57:55.308454 4775 ts_manager.cc:194] Registered new tserver with Master: 6e87defbedd749738d8fd9485c9c345f (127.4.158.3:38573)
I20250812 13:57:55.309648 4775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:42967
W20250812 13:57:55.597424 5233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:57:55.597941 5233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:57:55.598430 5233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:57:55.631549 5233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:57:55.632366 5233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.4
I20250812 13:57:55.666596 5233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.4.158.4
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.4
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:57:55.667881 5233 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:57:55.669409 5233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:57:55.680217 5239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:57:56.312553 5229 heartbeater.cc:499] Master 127.4.158.62:46075 was elected leader, sending a full tablet report...
W20250812 13:57:55.681593 5240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:56.934170 5242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:56.936199 5241 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1250 milliseconds
W20250812 13:57:56.936398 5233 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.256s user 0.438s sys 0.817s
W20250812 13:57:56.936745 5233 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.256s user 0.438s sys 0.817s
I20250812 13:57:56.937002 5233 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:57:56.938098 5233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:57:56.942459 5233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:57:56.943858 5233 hybrid_clock.cc:648] HybridClock initialized: now 1755007076943798 us; error 69 us; skew 500 ppm
I20250812 13:57:56.944636 5233 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:57:56.951627 5233 webserver.cc:480] Webserver started at http://127.4.158.4:33329/ using document root <none> and password file <none>
I20250812 13:57:56.952507 5233 fs_manager.cc:362] Metadata directory not provided
I20250812 13:57:56.952701 5233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:57:56.953145 5233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:57:56.957473 5233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "af2b1d0f99a94941b5097a11fa0d8026"
format_stamp: "Formatted at 2025-08-12 13:57:56 on dist-test-slave-f6np"
I20250812 13:57:56.958609 5233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "af2b1d0f99a94941b5097a11fa0d8026"
format_stamp: "Formatted at 2025-08-12 13:57:56 on dist-test-slave-f6np"
I20250812 13:57:56.966409 5233 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20250812 13:57:56.972383 5250 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:56.973534 5233 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250812 13:57:56.973834 5233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "af2b1d0f99a94941b5097a11fa0d8026"
format_stamp: "Formatted at 2025-08-12 13:57:56 on dist-test-slave-f6np"
I20250812 13:57:56.974175 5233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:57:57.043495 5233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:57:57.044940 5233 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:57:57.045358 5233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:57:57.047875 5233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:57:57.051915 5233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:57:57.052109 5233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:57.052337 5233 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:57:57.052486 5233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:57.178493 5233 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.4:33725
I20250812 13:57:57.178584 5362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.4:33725 every 8 connection(s)
I20250812 13:57:57.180955 5233 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250812 13:57:57.183149 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 5233
I20250812 13:57:57.183866 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250812 13:57:57.204946 5363 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46075
I20250812 13:57:57.205372 5363 heartbeater.cc:461] Registering TS with master...
I20250812 13:57:57.206349 5363 heartbeater.cc:507] Master 127.4.158.62:46075 requested a full tablet report, sending...
I20250812 13:57:57.208281 4775 ts_manager.cc:194] Registered new tserver with Master: af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:57:57.209481 4775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.4:48399
I20250812 13:57:57.218454 4728 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250812 13:57:57.257727 4775 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41090:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250812 13:57:57.326403 5164 tablet_service.cc:1468] Processing CreateTablet for tablet 34e527f01fd44932b437b6516a392541 (DEFAULT_TABLE table=TestTable [id=adc00b3a9b1a4e7ea3f9edc8ad88b54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:57:57.328459 5164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e527f01fd44932b437b6516a392541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:57:57.333981 5031 tablet_service.cc:1468] Processing CreateTablet for tablet 34e527f01fd44932b437b6516a392541 (DEFAULT_TABLE table=TestTable [id=adc00b3a9b1a4e7ea3f9edc8ad88b54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:57:57.335664 5031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e527f01fd44932b437b6516a392541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:57:57.335330 5298 tablet_service.cc:1468] Processing CreateTablet for tablet 34e527f01fd44932b437b6516a392541 (DEFAULT_TABLE table=TestTable [id=adc00b3a9b1a4e7ea3f9edc8ad88b54e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:57:57.337213 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e527f01fd44932b437b6516a392541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:57:57.357993 5382 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3: Bootstrap starting.
I20250812 13:57:57.360498 5383 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f: Bootstrap starting.
I20250812 13:57:57.363765 5384 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: Bootstrap starting.
I20250812 13:57:57.365929 5382 tablet_bootstrap.cc:654] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3: Neither blocks nor log segments found. Creating new log.
I20250812 13:57:57.367846 5383 tablet_bootstrap.cc:654] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f: Neither blocks nor log segments found. Creating new log.
I20250812 13:57:57.368377 5382 log.cc:826] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3: Log is configured to *not* fsync() on all Append() calls
I20250812 13:57:57.369562 5383 log.cc:826] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f: Log is configured to *not* fsync() on all Append() calls
I20250812 13:57:57.372130 5384 tablet_bootstrap.cc:654] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: Neither blocks nor log segments found. Creating new log.
I20250812 13:57:57.374054 5382 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3: No bootstrap required, opened a new log
I20250812 13:57:57.374305 5384 log.cc:826] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: Log is configured to *not* fsync() on all Append() calls
I20250812 13:57:57.374563 5382 ts_tablet_manager.cc:1397] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20250812 13:57:57.375487 5383 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f: No bootstrap required, opened a new log
I20250812 13:57:57.376009 5383 ts_tablet_manager.cc:1397] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250812 13:57:57.379376 5384 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: No bootstrap required, opened a new log
I20250812 13:57:57.379878 5384 ts_tablet_manager.cc:1397] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20250812 13:57:57.394131 5383 raft_consensus.cc:357] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.394810 5383 raft_consensus.cc:383] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:57:57.395061 5383 raft_consensus.cc:738] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e87defbedd749738d8fd9485c9c345f, State: Initialized, Role: FOLLOWER
I20250812 13:57:57.395790 5383 consensus_queue.cc:260] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.399718 5383 ts_tablet_manager.cc:1428] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f: Time spent starting tablet: real 0.023s user 0.014s sys 0.007s
I20250812 13:57:57.400941 5382 raft_consensus.cc:357] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.401875 5382 raft_consensus.cc:383] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:57:57.402168 5382 raft_consensus.cc:738] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8deec1959abc41ee89fdab731a18ede3, State: Initialized, Role: FOLLOWER
I20250812 13:57:57.403009 5382 consensus_queue.cc:260] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.405210 5384 raft_consensus.cc:357] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.406150 5384 raft_consensus.cc:383] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:57:57.406497 5384 raft_consensus.cc:738] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2b1d0f99a94941b5097a11fa0d8026, State: Initialized, Role: FOLLOWER
I20250812 13:57:57.407187 5382 ts_tablet_manager.cc:1428] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3: Time spent starting tablet: real 0.032s user 0.023s sys 0.009s
I20250812 13:57:57.407387 5384 consensus_queue.cc:260] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.410723 5363 heartbeater.cc:499] Master 127.4.158.62:46075 was elected leader, sending a full tablet report...
I20250812 13:57:57.412809 5384 ts_tablet_manager.cc:1428] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: Time spent starting tablet: real 0.033s user 0.026s sys 0.008s
I20250812 13:57:57.419804 5389 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:57:57.420246 5389 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.422605 5389 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e87defbedd749738d8fd9485c9c345f (127.4.158.3:38573), af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:57:57.424211 5388 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:57:57.424649 5388 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
W20250812 13:57:57.436399 5364 tablet.cc:2378] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:57:57.439544 5318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "8deec1959abc41ee89fdab731a18ede3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2b1d0f99a94941b5097a11fa0d8026" is_pre_election: true
I20250812 13:57:57.440431 5318 raft_consensus.cc:2466] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8deec1959abc41ee89fdab731a18ede3 in term 0.
I20250812 13:57:57.441880 4985 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8deec1959abc41ee89fdab731a18ede3, af2b1d0f99a94941b5097a11fa0d8026; no voters:
I20250812 13:57:57.442800 5389 raft_consensus.cc:2802] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 13:57:57.443151 5389 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:57:57.443471 5389 raft_consensus.cc:3058] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:57:57.443773 5051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "6e87defbedd749738d8fd9485c9c345f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8deec1959abc41ee89fdab731a18ede3" is_pre_election: true
I20250812 13:57:57.446372 5388 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397), af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:57:57.446843 5184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "8deec1959abc41ee89fdab731a18ede3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e87defbedd749738d8fd9485c9c345f" is_pre_election: true
I20250812 13:57:57.447575 5184 raft_consensus.cc:2466] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8deec1959abc41ee89fdab731a18ede3 in term 0.
I20250812 13:57:57.449338 5318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "6e87defbedd749738d8fd9485c9c345f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2b1d0f99a94941b5097a11fa0d8026" is_pre_election: true
I20250812 13:57:57.449736 5318 raft_consensus.cc:2466] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e87defbedd749738d8fd9485c9c345f in term 0.
I20250812 13:57:57.449836 5389 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.450707 5118 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e87defbedd749738d8fd9485c9c345f, af2b1d0f99a94941b5097a11fa0d8026; no voters:
I20250812 13:57:57.450968 5051 raft_consensus.cc:2391] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e87defbedd749738d8fd9485c9c345f in current term 1: Already voted for candidate 8deec1959abc41ee89fdab731a18ede3 in this term.
I20250812 13:57:57.451699 5389 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [CANDIDATE]: Term 1 election: Requested vote from peers 6e87defbedd749738d8fd9485c9c345f (127.4.158.3:38573), af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:57:57.452143 5388 raft_consensus.cc:2802] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 13:57:57.452476 5388 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:57:57.452714 5388 raft_consensus.cc:3058] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:57:57.452592 5318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "8deec1959abc41ee89fdab731a18ede3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2b1d0f99a94941b5097a11fa0d8026"
I20250812 13:57:57.453083 5318 raft_consensus.cc:3058] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:57:57.457760 5184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "8deec1959abc41ee89fdab731a18ede3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e87defbedd749738d8fd9485c9c345f"
I20250812 13:57:57.457902 5388 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.458966 5184 raft_consensus.cc:2391] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8deec1959abc41ee89fdab731a18ede3 in current term 1: Already voted for candidate 6e87defbedd749738d8fd9485c9c345f in this term.
I20250812 13:57:57.459544 5318 raft_consensus.cc:2466] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8deec1959abc41ee89fdab731a18ede3 in term 1.
I20250812 13:57:57.459618 5388 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 1 election: Requested vote from peers 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397), af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:57:57.460682 5318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "6e87defbedd749738d8fd9485c9c345f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2b1d0f99a94941b5097a11fa0d8026"
I20250812 13:57:57.460572 4985 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8deec1959abc41ee89fdab731a18ede3, af2b1d0f99a94941b5097a11fa0d8026; no voters: 6e87defbedd749738d8fd9485c9c345f
I20250812 13:57:57.461344 5318 raft_consensus.cc:2391] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e87defbedd749738d8fd9485c9c345f in current term 1: Already voted for candidate 8deec1959abc41ee89fdab731a18ede3 in this term.
I20250812 13:57:57.460623 5051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "6e87defbedd749738d8fd9485c9c345f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8deec1959abc41ee89fdab731a18ede3"
I20250812 13:57:57.461427 5389 raft_consensus.cc:2802] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:57:57.463009 5389 raft_consensus.cc:695] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [term 1 LEADER]: Becoming Leader. State: Replica: 8deec1959abc41ee89fdab731a18ede3, State: Running, Role: LEADER
I20250812 13:57:57.463781 5389 consensus_queue.cc:237] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:57.471419 5119 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e87defbedd749738d8fd9485c9c345f; no voters: 8deec1959abc41ee89fdab731a18ede3, af2b1d0f99a94941b5097a11fa0d8026
I20250812 13:57:57.472150 5388 raft_consensus.cc:2747] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250812 13:57:57.473771 4774 catalog_manager.cc:5582] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2). New cstate: current_term: 1 leader_uuid: "8deec1959abc41ee89fdab731a18ede3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } health_report { overall_health: UNKNOWN } } }
I20250812 13:57:57.496810 4728 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250812 13:57:57.500140 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8deec1959abc41ee89fdab731a18ede3 to finish bootstrapping
I20250812 13:57:57.513399 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6e87defbedd749738d8fd9485c9c345f to finish bootstrapping
I20250812 13:57:57.523692 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver af2b1d0f99a94941b5097a11fa0d8026 to finish bootstrapping
I20250812 13:57:57.534093 4728 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
W20250812 13:57:57.540851 5230 tablet.cc:2378] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:57:57.537328 4728 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "34e527f01fd44932b437b6516a392541"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "6e87defbedd749738d8fd9485c9c345f"
rpc_addresses {
host: "127.4.158.3"
port: 38573
}
}
ts_infos {
permanent_uuid: "8deec1959abc41ee89fdab731a18ede3"
rpc_addresses {
host: "127.4.158.2"
port: 44397
}
}
ts_infos {
permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026"
rpc_addresses {
host: "127.4.158.4"
port: 33725
}
}
W20250812 13:57:57.588205 5097 tablet.cc:2378] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:57:57.973289 5400 consensus_queue.cc:1035] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 13:57:57.989709 5400 consensus_queue.cc:1035] T 34e527f01fd44932b437b6516a392541 P 8deec1959abc41ee89fdab731a18ede3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 13:57:57.991943 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 4967
W20250812 13:57:58.012180 4759 connection.cc:537] server connection from 127.4.158.2:60677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250812 13:57:58.012739 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 4742
I20250812 13:57:58.039554 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:46075
--webserver_interface=127.4.158.62
--webserver_port=34381
--builtin_ntp_servers=127.4.158.20:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:46075 with env {}
W20250812 13:57:58.332427 5407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:57:58.333020 5407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:57:58.333473 5407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:57:58.364261 5407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:57:58.364598 5407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:57:58.364847 5407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:57:58.365088 5407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:57:58.399993 5407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:46075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=34381
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:57:58.401302 5407 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:57:58.402904 5407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:57:58.415381 5414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:58.426072 5229 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:46075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:46075: connect: Connection refused (error 111)
W20250812 13:57:58.924150 4963 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:46075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:46075: connect: Connection refused (error 111)
W20250812 13:57:59.015268 5363 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:46075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:46075: connect: Connection refused (error 111)
I20250812 13:57:59.405463 5422 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:57:59.406333 5422 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:59.412664 5422 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397), af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:57:59.412544 5318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "6e87defbedd749738d8fd9485c9c345f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2b1d0f99a94941b5097a11fa0d8026" is_pre_election: true
W20250812 13:57:59.413794 5119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.158.2:44397: connect: Connection refused (error 111)
W20250812 13:57:59.427141 5119 leader_election.cc:336] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397): Network error: Client connection negotiation failed: client connection to 127.4.158.2:44397: connect: Connection refused (error 111)
I20250812 13:57:59.427682 5119 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e87defbedd749738d8fd9485c9c345f; no voters: 8deec1959abc41ee89fdab731a18ede3, af2b1d0f99a94941b5097a11fa0d8026
I20250812 13:57:59.428510 5422 raft_consensus.cc:2747] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250812 13:57:59.500413 5425 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8deec1959abc41ee89fdab731a18ede3)
I20250812 13:57:59.501237 5425 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:59.514966 5425 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e87defbedd749738d8fd9485c9c345f (127.4.158.3:38573), 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397)
I20250812 13:57:59.523883 5184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "af2b1d0f99a94941b5097a11fa0d8026" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e87defbedd749738d8fd9485c9c345f" is_pre_election: true
I20250812 13:57:59.524537 5184 raft_consensus.cc:2466] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af2b1d0f99a94941b5097a11fa0d8026 in term 1.
I20250812 13:57:59.526101 5252 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e87defbedd749738d8fd9485c9c345f, af2b1d0f99a94941b5097a11fa0d8026; no voters:
I20250812 13:57:59.527108 5425 raft_consensus.cc:2802] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250812 13:57:59.527514 5425 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8deec1959abc41ee89fdab731a18ede3)
I20250812 13:57:59.527897 5425 raft_consensus.cc:3058] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 1 FOLLOWER]: Advancing to term 2
W20250812 13:57:59.530917 5253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.158.2:44397: connect: Connection refused (error 111)
W20250812 13:57:59.541041 5253 leader_election.cc:336] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397): Network error: Client connection negotiation failed: client connection to 127.4.158.2:44397: connect: Connection refused (error 111)
I20250812 13:57:59.542084 5425 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:57:59.545080 5184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "af2b1d0f99a94941b5097a11fa0d8026" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e87defbedd749738d8fd9485c9c345f"
I20250812 13:57:59.545693 5184 raft_consensus.cc:3058] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 1 FOLLOWER]: Advancing to term 2
I20250812 13:57:59.554370 5184 raft_consensus.cc:2466] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af2b1d0f99a94941b5097a11fa0d8026 in term 2.
I20250812 13:57:59.555721 5252 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e87defbedd749738d8fd9485c9c345f, af2b1d0f99a94941b5097a11fa0d8026; no voters:
I20250812 13:57:59.558359 5429 raft_consensus.cc:2802] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 13:57:59.560005 5425 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 2 election: Requested vote from peers 6e87defbedd749738d8fd9485c9c345f (127.4.158.3:38573), 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397)
W20250812 13:57:59.565383 5253 leader_election.cc:336] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397): Network error: Client connection negotiation failed: client connection to 127.4.158.2:44397: connect: Connection refused (error 111)
I20250812 13:57:59.566607 5429 raft_consensus.cc:695] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 2 LEADER]: Becoming Leader. State: Replica: af2b1d0f99a94941b5097a11fa0d8026, State: Running, Role: LEADER
I20250812 13:57:59.567368 5429 consensus_queue.cc:237] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
W20250812 13:57:58.416392 5413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:59.942174 5407 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.528s user 0.537s sys 0.980s
W20250812 13:57:59.942616 5407 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.529s user 0.537s sys 0.981s
W20250812 13:57:59.816887 5412 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 5407
W20250812 13:57:59.944352 5416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:57:59.946960 5415 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1533 milliseconds
I20250812 13:57:59.947024 5407 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:57:59.948134 5407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:57:59.950511 5407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:57:59.951839 5407 hybrid_clock.cc:648] HybridClock initialized: now 1755007079951796 us; error 48 us; skew 500 ppm
I20250812 13:57:59.952603 5407 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:57:59.958398 5407 webserver.cc:480] Webserver started at http://127.4.158.62:34381/ using document root <none> and password file <none>
I20250812 13:57:59.959278 5407 fs_manager.cc:362] Metadata directory not provided
I20250812 13:57:59.959504 5407 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:57:59.966948 5407 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s
I20250812 13:57:59.971181 5437 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:57:59.972112 5407 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250812 13:57:59.972404 5407 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "36fb135f8c044dd48cf762a42bc2ff74"
format_stamp: "Formatted at 2025-08-12 13:57:49 on dist-test-slave-f6np"
I20250812 13:57:59.974287 5407 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250812 13:57:59.989696 5253 consensus_peers.cc:489] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 -> Peer 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397): Couldn't send request to peer 8deec1959abc41ee89fdab731a18ede3. Status: Network error: Client connection negotiation failed: client connection to 127.4.158.2:44397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250812 13:58:00.037376 5184 raft_consensus.cc:1273] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 2 FOLLOWER]: Refusing update from remote peer af2b1d0f99a94941b5097a11fa0d8026: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250812 13:58:00.037475 5407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:00.038582 5429 consensus_queue.cc:1035] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 13:58:00.039031 5407 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:00.039445 5407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:00.111075 5407 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:46075
I20250812 13:58:00.111192 5494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:46075 every 8 connection(s)
I20250812 13:58:00.113885 5407 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:58:00.123490 5495 sys_catalog.cc:263] Verifying existing consensus state
I20250812 13:58:00.124410 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 5407
I20250812 13:58:00.124868 4728 kudu-admin-test.cc:735] Forcing unsafe config change on tserver af2b1d0f99a94941b5097a11fa0d8026
I20250812 13:58:00.128180 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Bootstrap starting.
I20250812 13:58:00.178524 5495 log.cc:826] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:00.207854 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:58:00.208853 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Bootstrap complete.
I20250812 13:58:00.230198 5495 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } }
I20250812 13:58:00.232354 5495 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fb135f8c044dd48cf762a42bc2ff74, State: Initialized, Role: FOLLOWER
I20250812 13:58:00.233126 5495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } }
I20250812 13:58:00.233675 5495 raft_consensus.cc:397] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:58:00.233958 5495 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:58:00.234257 5495 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 1 FOLLOWER]: Advancing to term 2
I20250812 13:58:00.239953 5495 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } }
I20250812 13:58:00.240650 5495 leader_election.cc:304] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36fb135f8c044dd48cf762a42bc2ff74; no voters:
I20250812 13:58:00.242380 5495 leader_election.cc:290] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 13:58:00.242997 5499 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 13:58:00.246726 5499 raft_consensus.cc:695] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [term 2 LEADER]: Becoming Leader. State: Replica: 36fb135f8c044dd48cf762a42bc2ff74, State: Running, Role: LEADER
I20250812 13:58:00.246780 5495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:58:00.247648 5499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } }
I20250812 13:58:00.258038 5501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36fb135f8c044dd48cf762a42bc2ff74. Latest consensus state: current_term: 2 leader_uuid: "36fb135f8c044dd48cf762a42bc2ff74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } } }
I20250812 13:58:00.258744 5501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:00.260751 5500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "36fb135f8c044dd48cf762a42bc2ff74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fb135f8c044dd48cf762a42bc2ff74" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46075 } } }
I20250812 13:58:00.261441 5500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:00.272241 5507 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:58:00.311203 5507 catalog_manager.cc:671] Loaded metadata for table TestTable [id=adc00b3a9b1a4e7ea3f9edc8ad88b54e]
I20250812 13:58:00.319258 5507 tablet_loader.cc:96] loaded metadata for tablet 34e527f01fd44932b437b6516a392541 (table TestTable [id=adc00b3a9b1a4e7ea3f9edc8ad88b54e])
I20250812 13:58:00.320961 5507 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:58:00.325728 5507 catalog_manager.cc:1261] Loaded cluster ID: 7501bfc9943d477aa85de342d29f7225
I20250812 13:58:00.326112 5507 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:58:00.330796 5516 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250812 13:58:00.334050 5507 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:58:00.339236 5507 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 36fb135f8c044dd48cf762a42bc2ff74: Loaded TSK: 0
I20250812 13:58:00.340842 5507 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250812 13:58:00.486450 5497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:00.487033 5497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:00.517598 5497 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250812 13:58:00.980209 4963 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46075
I20250812 13:58:00.985949 5460 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" instance_seqno: 1755007071792187) as {username='slave'} at 127.4.158.1:46161; Asking this server to re-register.
I20250812 13:58:00.987762 4963 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:00.988534 4963 heartbeater.cc:507] Master 127.4.158.62:46075 requested a full tablet report, sending...
I20250812 13:58:00.991250 5459 ts_manager.cc:194] Registered new tserver with Master: ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1:35893)
I20250812 13:58:01.091320 5363 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46075
I20250812 13:58:01.095738 5459 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" instance_seqno: 1755007077149658) as {username='slave'} at 127.4.158.4:50567; Asking this server to re-register.
I20250812 13:58:01.097506 5363 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:01.098259 5363 heartbeater.cc:507] Master 127.4.158.62:46075 requested a full tablet report, sending...
I20250812 13:58:01.101500 5459 ts_manager.cc:194] Registered new tserver with Master: af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:58:01.105981 5459 catalog_manager.cc:5582] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 reported cstate change: term changed from 1 to 2, leader changed from 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2) to af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4). New cstate: current_term: 2 leader_uuid: "af2b1d0f99a94941b5097a11fa0d8026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } health_report { overall_health: HEALTHY } } }
I20250812 13:58:01.130565 5229 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46075
I20250812 13:58:01.134377 5460 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" instance_seqno: 1755007075252814) as {username='slave'} at 127.4.158.3:56087; Asking this server to re-register.
I20250812 13:58:01.136317 5229 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:01.137037 5229 heartbeater.cc:507] Master 127.4.158.62:46075 requested a full tablet report, sending...
I20250812 13:58:01.140303 5460 ts_manager.cc:194] Registered new tserver with Master: 6e87defbedd749738d8fd9485c9c345f (127.4.158.3:38573)
W20250812 13:58:01.881683 5497 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.325s user 0.520s sys 0.779s
W20250812 13:58:01.882016 5497 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.325s user 0.524s sys 0.779s
I20250812 13:58:01.945065 5318 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "af2b1d0f99a94941b5097a11fa0d8026"
tablet_id: "34e527f01fd44932b437b6516a392541"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "6e87defbedd749738d8fd9485c9c345f"
}
peers {
permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026"
}
}
from {username='slave'} at 127.0.0.1:54262
W20250812 13:58:01.946524 5318 raft_consensus.cc:2216] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } unsafe_config_change: true
I20250812 13:58:01.947683 5318 raft_consensus.cc:3053] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 2 LEADER]: Stepping down as leader of term 2
I20250812 13:58:01.947978 5318 raft_consensus.cc:738] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: af2b1d0f99a94941b5097a11fa0d8026, State: Running, Role: LEADER
I20250812 13:58:01.948807 5318 consensus_queue.cc:260] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:58:01.950470 5318 raft_consensus.cc:3058] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 2 FOLLOWER]: Advancing to term 3
I20250812 13:58:03.388315 5545 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader af2b1d0f99a94941b5097a11fa0d8026)
I20250812 13:58:03.388708 5545 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } }
I20250812 13:58:03.390110 5545 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397), af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:58:03.391330 5318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "6e87defbedd749738d8fd9485c9c345f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "af2b1d0f99a94941b5097a11fa0d8026" is_pre_election: true
W20250812 13:58:03.394789 5119 leader_election.cc:336] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2:44397): Network error: Client connection negotiation failed: client connection to 127.4.158.2:44397: connect: Connection refused (error 111)
I20250812 13:58:03.395089 5119 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e87defbedd749738d8fd9485c9c345f; no voters: 8deec1959abc41ee89fdab731a18ede3, af2b1d0f99a94941b5097a11fa0d8026
I20250812 13:58:03.395567 5545 raft_consensus.cc:2747] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250812 13:58:03.458482 5548 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250812 13:58:03.458869 5548 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } unsafe_config_change: true
I20250812 13:58:03.459913 5548 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e87defbedd749738d8fd9485c9c345f (127.4.158.3:38573)
I20250812 13:58:03.460937 5184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "af2b1d0f99a94941b5097a11fa0d8026" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e87defbedd749738d8fd9485c9c345f" is_pre_election: true
I20250812 13:58:03.461432 5184 raft_consensus.cc:2466] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af2b1d0f99a94941b5097a11fa0d8026 in term 2.
I20250812 13:58:03.462453 5252 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6e87defbedd749738d8fd9485c9c345f, af2b1d0f99a94941b5097a11fa0d8026; no voters:
I20250812 13:58:03.463102 5548 raft_consensus.cc:2802] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250812 13:58:03.463371 5548 raft_consensus.cc:491] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250812 13:58:03.463626 5548 raft_consensus.cc:3058] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 3 FOLLOWER]: Advancing to term 4
I20250812 13:58:03.467726 5548 raft_consensus.cc:513] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } unsafe_config_change: true
I20250812 13:58:03.468860 5548 leader_election.cc:290] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 4 election: Requested vote from peers 6e87defbedd749738d8fd9485c9c345f (127.4.158.3:38573)
I20250812 13:58:03.469884 5184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34e527f01fd44932b437b6516a392541" candidate_uuid: "af2b1d0f99a94941b5097a11fa0d8026" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e87defbedd749738d8fd9485c9c345f"
I20250812 13:58:03.470304 5184 raft_consensus.cc:3058] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 2 FOLLOWER]: Advancing to term 4
I20250812 13:58:03.474321 5184 raft_consensus.cc:2466] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af2b1d0f99a94941b5097a11fa0d8026 in term 4.
I20250812 13:58:03.475221 5252 leader_election.cc:304] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6e87defbedd749738d8fd9485c9c345f, af2b1d0f99a94941b5097a11fa0d8026; no voters:
I20250812 13:58:03.475808 5548 raft_consensus.cc:2802] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 4 FOLLOWER]: Leader election won for term 4
I20250812 13:58:03.476550 5548 raft_consensus.cc:695] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 4 LEADER]: Becoming Leader. State: Replica: af2b1d0f99a94941b5097a11fa0d8026, State: Running, Role: LEADER
I20250812 13:58:03.477188 5548 consensus_queue.cc:237] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } unsafe_config_change: true
I20250812 13:58:03.482473 5460 catalog_manager.cc:5582] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 6e87defbedd749738d8fd9485c9c345f (127.4.158.3), VOTER af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4). New cstate: current_term: 4 leader_uuid: "af2b1d0f99a94941b5097a11fa0d8026" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8deec1959abc41ee89fdab731a18ede3" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 44397 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } unsafe_config_change: true }
I20250812 13:58:03.988704 5184 raft_consensus.cc:1273] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 4 FOLLOWER]: Refusing update from remote peer af2b1d0f99a94941b5097a11fa0d8026: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250812 13:58:03.990217 5556 consensus_queue.cc:1035] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250812 13:58:04.000236 5559 raft_consensus.cc:2953] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } unsafe_config_change: true }
I20250812 13:58:04.002462 5184 raft_consensus.cc:2953] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } unsafe_config_change: true }
I20250812 13:58:04.013231 5460 catalog_manager.cc:5582] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 reported cstate change: config changed from index -1 to 3, VOTER 8deec1959abc41ee89fdab731a18ede3 (127.4.158.2) evicted, no longer has a pending config: VOTER 6e87defbedd749738d8fd9485c9c345f (127.4.158.3), VOTER af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4). New cstate: current_term: 4 leader_uuid: "af2b1d0f99a94941b5097a11fa0d8026" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250812 13:58:04.022200 5460 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 34e527f01fd44932b437b6516a392541 on TS 8deec1959abc41ee89fdab731a18ede3: Not found: failed to reset TS proxy: Could not find TS for UUID 8deec1959abc41ee89fdab731a18ede3
I20250812 13:58:04.038400 5318 consensus_queue.cc:237] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: true } } unsafe_config_change: true
I20250812 13:58:04.042902 5184 raft_consensus.cc:1273] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 4 FOLLOWER]: Refusing update from remote peer af2b1d0f99a94941b5097a11fa0d8026: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250812 13:58:04.044253 5559 consensus_queue.cc:1035] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250812 13:58:04.049918 5556 raft_consensus.cc:2953] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: true } } unsafe_config_change: true }
I20250812 13:58:04.050959 5184 raft_consensus.cc:2953] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: true } } unsafe_config_change: true }
W20250812 13:58:04.053928 5251 consensus_peers.cc:489] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 -> Peer ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1:35893): Couldn't send request to peer ab9c948b6f0e4ba98971df43318d7228. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34e527f01fd44932b437b6516a392541. This is attempt 1: this message will repeat every 5th retry.
I20250812 13:58:04.057840 5439 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 34e527f01fd44932b437b6516a392541 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250812 13:58:04.060220 5459 catalog_manager.cc:5582] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f reported cstate change: config changed from index 3 to 5, NON_VOTER ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1) added. New cstate: current_term: 4 leader_uuid: "af2b1d0f99a94941b5097a11fa0d8026" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: true } } unsafe_config_change: true }
W20250812 13:58:04.075014 5439 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 34e527f01fd44932b437b6516a392541 on TS 8deec1959abc41ee89fdab731a18ede3 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8deec1959abc41ee89fdab731a18ede3
I20250812 13:58:04.669266 5569 ts_tablet_manager.cc:927] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: Initiating tablet copy from peer af2b1d0f99a94941b5097a11fa0d8026 (127.4.158.4:33725)
I20250812 13:58:04.671396 5569 tablet_copy_client.cc:323] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: tablet copy: Beginning tablet copy session from remote peer at address 127.4.158.4:33725
I20250812 13:58:04.681258 5338 tablet_copy_service.cc:140] P af2b1d0f99a94941b5097a11fa0d8026: Received BeginTabletCopySession request for tablet 34e527f01fd44932b437b6516a392541 from peer ab9c948b6f0e4ba98971df43318d7228 ({username='slave'} at 127.4.158.1:33943)
I20250812 13:58:04.681635 5338 tablet_copy_service.cc:161] P af2b1d0f99a94941b5097a11fa0d8026: Beginning new tablet copy session on tablet 34e527f01fd44932b437b6516a392541 from peer ab9c948b6f0e4ba98971df43318d7228 at {username='slave'} at 127.4.158.1:33943: session id = ab9c948b6f0e4ba98971df43318d7228-34e527f01fd44932b437b6516a392541
I20250812 13:58:04.686116 5338 tablet_copy_source_session.cc:215] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 13:58:04.690891 5569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34e527f01fd44932b437b6516a392541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:04.710042 5569 tablet_copy_client.cc:806] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: tablet copy: Starting download of 0 data blocks...
I20250812 13:58:04.710618 5569 tablet_copy_client.cc:670] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: tablet copy: Starting download of 1 WAL segments...
I20250812 13:58:04.714030 5569 tablet_copy_client.cc:538] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 13:58:04.719566 5569 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: Bootstrap starting.
I20250812 13:58:04.730698 5569 log.cc:826] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:04.741236 5569 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:58:04.741914 5569 tablet_bootstrap.cc:492] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: Bootstrap complete.
I20250812 13:58:04.742436 5569 ts_tablet_manager.cc:1397] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.004s
I20250812 13:58:04.759474 5569 raft_consensus.cc:357] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: true } } unsafe_config_change: true
I20250812 13:58:04.760289 5569 raft_consensus.cc:738] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: ab9c948b6f0e4ba98971df43318d7228, State: Initialized, Role: LEARNER
I20250812 13:58:04.760895 5569 consensus_queue.cc:260] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: true } } unsafe_config_change: true
I20250812 13:58:04.763836 5569 ts_tablet_manager.cc:1428] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228: Time spent starting tablet: real 0.021s user 0.017s sys 0.004s
I20250812 13:58:04.765319 5338 tablet_copy_service.cc:342] P af2b1d0f99a94941b5097a11fa0d8026: Request end of tablet copy session ab9c948b6f0e4ba98971df43318d7228-34e527f01fd44932b437b6516a392541 received from {username='slave'} at 127.4.158.1:33943
I20250812 13:58:04.765653 5338 tablet_copy_service.cc:434] P af2b1d0f99a94941b5097a11fa0d8026: ending tablet copy session ab9c948b6f0e4ba98971df43318d7228-34e527f01fd44932b437b6516a392541 on tablet 34e527f01fd44932b437b6516a392541 with peer ab9c948b6f0e4ba98971df43318d7228
I20250812 13:58:05.081770 4918 raft_consensus.cc:1215] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250812 13:58:05.241799 5439 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 34e527f01fd44932b437b6516a392541 on TS 8deec1959abc41ee89fdab731a18ede3 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8deec1959abc41ee89fdab731a18ede3
I20250812 13:58:05.687302 5576 raft_consensus.cc:1062] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026: attempting to promote NON_VOTER ab9c948b6f0e4ba98971df43318d7228 to VOTER
I20250812 13:58:05.689258 5576 consensus_queue.cc:237] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: false } } unsafe_config_change: true
I20250812 13:58:05.694235 4918 raft_consensus.cc:1273] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228 [term 4 LEARNER]: Refusing update from remote peer af2b1d0f99a94941b5097a11fa0d8026: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250812 13:58:05.694669 5184 raft_consensus.cc:1273] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 4 FOLLOWER]: Refusing update from remote peer af2b1d0f99a94941b5097a11fa0d8026: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250812 13:58:05.695441 5576 consensus_queue.cc:1035] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250812 13:58:05.696219 5580 consensus_queue.cc:1035] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250812 13:58:05.702075 5580 raft_consensus.cc:2953] T 34e527f01fd44932b437b6516a392541 P af2b1d0f99a94941b5097a11fa0d8026 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: false } } unsafe_config_change: true }
I20250812 13:58:05.703461 5184 raft_consensus.cc:2953] T 34e527f01fd44932b437b6516a392541 P 6e87defbedd749738d8fd9485c9c345f [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: false } } unsafe_config_change: true }
I20250812 13:58:05.704336 4918 raft_consensus.cc:2953] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: false } } unsafe_config_change: true }
I20250812 13:58:05.715569 5459 catalog_manager.cc:5582] T 34e527f01fd44932b437b6516a392541 P ab9c948b6f0e4ba98971df43318d7228 reported cstate change: config changed from index 5 to 6, ab9c948b6f0e4ba98971df43318d7228 (127.4.158.1) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "af2b1d0f99a94941b5097a11fa0d8026" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e87defbedd749738d8fd9485c9c345f" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38573 } } peers { permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 33725 } } peers { permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35893 } attrs { promote: false } } unsafe_config_change: true }
I20250812 13:58:05.796546 4728 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250812 13:58:05.808789 4728 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "34e527f01fd44932b437b6516a392541"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "6e87defbedd749738d8fd9485c9c345f"
rpc_addresses {
host: "127.4.158.3"
port: 38573
}
}
ts_infos {
permanent_uuid: "af2b1d0f99a94941b5097a11fa0d8026"
rpc_addresses {
host: "127.4.158.4"
port: 33725
}
}
ts_infos {
permanent_uuid: "ab9c948b6f0e4ba98971df43318d7228"
rpc_addresses {
host: "127.4.158.1"
port: 35893
}
}
I20250812 13:58:05.810890 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 4834
I20250812 13:58:05.830014 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 5100
I20250812 13:58:05.851122 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 5233
I20250812 13:58:05.871959 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 5407
2025-08-12T13:58:05Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19131 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250812 13:58:05.926427 4728 test_util.cc:276] Using random seed: 1844402178
I20250812 13:58:05.932044 4728 ts_itest-base.cc:115] Starting cluster with:
I20250812 13:58:05.932210 4728 ts_itest-base.cc:116] --------------
I20250812 13:58:05.932363 4728 ts_itest-base.cc:117] 3 tablet servers
I20250812 13:58:05.932524 4728 ts_itest-base.cc:118] 3 replicas per TS
I20250812 13:58:05.932691 4728 ts_itest-base.cc:119] --------------
2025-08-12T13:58:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T13:58:05Z Disabled control of system clock
I20250812 13:58:05.965212 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:34825
--webserver_interface=127.4.158.62
--webserver_port=0
--builtin_ntp_servers=127.4.158.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:34825
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250812 13:58:06.256520 5600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:06.257079 5600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:06.257665 5600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:06.288790 5600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:58:06.289084 5600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:06.289295 5600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:58:06.289489 5600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:58:06.324434 5600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:34825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:06.325687 5600 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:06.327253 5600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:06.338294 5606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:07.740680 5605 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 5600
W20250812 13:58:07.872866 5605 kernel_stack_watchdog.cc:198] Thread 5600 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 13:58:06.338451 5607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:07.873499 5600 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.536s user 0.678s sys 0.856s
W20250812 13:58:07.873968 5600 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.536s user 0.679s sys 0.857s
W20250812 13:58:07.875490 5609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:07.878409 5608 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1540 milliseconds
I20250812 13:58:07.878432 5600 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:07.879609 5600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:07.881956 5600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:07.883277 5600 hybrid_clock.cc:648] HybridClock initialized: now 1755007087883244 us; error 51 us; skew 500 ppm
I20250812 13:58:07.884027 5600 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:07.889626 5600 webserver.cc:480] Webserver started at http://127.4.158.62:40791/ using document root <none> and password file <none>
I20250812 13:58:07.890512 5600 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:07.890720 5600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:07.891177 5600 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:07.895406 5600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "84c18dd07b3b4cafaec989b6c228d194"
format_stamp: "Formatted at 2025-08-12 13:58:07 on dist-test-slave-f6np"
I20250812 13:58:07.896462 5600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "84c18dd07b3b4cafaec989b6c228d194"
format_stamp: "Formatted at 2025-08-12 13:58:07 on dist-test-slave-f6np"
I20250812 13:58:07.903319 5600 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20250812 13:58:07.908578 5617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:07.909588 5600 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250812 13:58:07.909937 5600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "84c18dd07b3b4cafaec989b6c228d194"
format_stamp: "Formatted at 2025-08-12 13:58:07 on dist-test-slave-f6np"
I20250812 13:58:07.910243 5600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:07.961925 5600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:07.963316 5600 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:07.963739 5600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:08.029237 5600 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:34825
I20250812 13:58:08.029352 5668 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:34825 every 8 connection(s)
I20250812 13:58:08.031919 5600 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:58:08.036867 5669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:08.042142 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 5600
I20250812 13:58:08.042620 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 13:58:08.061730 5669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194: Bootstrap starting.
I20250812 13:58:08.068053 5669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:08.070334 5669 log.cc:826] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:08.074925 5669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194: No bootstrap required, opened a new log
I20250812 13:58:08.092869 5669 raft_consensus.cc:357] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c18dd07b3b4cafaec989b6c228d194" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 34825 } }
I20250812 13:58:08.093570 5669 raft_consensus.cc:383] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:08.093815 5669 raft_consensus.cc:738] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c18dd07b3b4cafaec989b6c228d194, State: Initialized, Role: FOLLOWER
I20250812 13:58:08.094524 5669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c18dd07b3b4cafaec989b6c228d194" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 34825 } }
I20250812 13:58:08.095034 5669 raft_consensus.cc:397] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:58:08.095311 5669 raft_consensus.cc:491] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:58:08.095624 5669 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:08.099645 5669 raft_consensus.cc:513] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c18dd07b3b4cafaec989b6c228d194" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 34825 } }
I20250812 13:58:08.100392 5669 leader_election.cc:304] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84c18dd07b3b4cafaec989b6c228d194; no voters:
I20250812 13:58:08.102550 5669 leader_election.cc:290] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:58:08.103364 5674 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:08.106492 5674 raft_consensus.cc:695] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [term 1 LEADER]: Becoming Leader. State: Replica: 84c18dd07b3b4cafaec989b6c228d194, State: Running, Role: LEADER
I20250812 13:58:08.107364 5674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c18dd07b3b4cafaec989b6c228d194" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 34825 } }
I20250812 13:58:08.110013 5669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:58:08.116276 5675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84c18dd07b3b4cafaec989b6c228d194. Latest consensus state: current_term: 1 leader_uuid: "84c18dd07b3b4cafaec989b6c228d194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c18dd07b3b4cafaec989b6c228d194" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 34825 } } }
I20250812 13:58:08.117681 5675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:08.117763 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84c18dd07b3b4cafaec989b6c228d194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84c18dd07b3b4cafaec989b6c228d194" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 34825 } } }
I20250812 13:58:08.118459 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:08.120522 5681 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:58:08.131880 5681 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:58:08.148568 5681 catalog_manager.cc:1349] Generated new cluster ID: d5cf022e5c3f4f4c8ea4841513081823
I20250812 13:58:08.148916 5681 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:58:08.166754 5681 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 13:58:08.168190 5681 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:58:08.179690 5681 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 84c18dd07b3b4cafaec989b6c228d194: Generated new TSK 0
I20250812 13:58:08.180517 5681 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 13:58:08.197633 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:0
--local_ip_for_outbound_sockets=127.4.158.1
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:34825
--builtin_ntp_servers=127.4.158.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250812 13:58:08.497239 5693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250812 13:58:08.497941 5693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:08.498200 5693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:08.498696 5693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:08.534679 5693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:08.535573 5693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 13:58:08.571610 5693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:34825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:08.572921 5693 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:08.574669 5693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:08.586941 5699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:09.939281 5701 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250812 13:58:08.589919 5700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:09.941192 5702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:09.944664 5693 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.357s user 0.571s sys 0.781s
W20250812 13:58:09.945062 5693 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.358s user 0.572s sys 0.783s
I20250812 13:58:09.945367 5693 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:09.950670 5693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:09.957787 5693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:09.959265 5693 hybrid_clock.cc:648] HybridClock initialized: now 1755007089959226 us; error 61 us; skew 500 ppm
I20250812 13:58:09.960125 5693 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:09.970645 5693 webserver.cc:480] Webserver started at http://127.4.158.1:46507/ using document root <none> and password file <none>
I20250812 13:58:09.971623 5693 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:09.971846 5693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:09.972329 5693 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:09.977074 5693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "63d8666af9b34b9499723813c60e1996"
format_stamp: "Formatted at 2025-08-12 13:58:09 on dist-test-slave-f6np"
I20250812 13:58:09.978299 5693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "63d8666af9b34b9499723813c60e1996"
format_stamp: "Formatted at 2025-08-12 13:58:09 on dist-test-slave-f6np"
I20250812 13:58:09.987638 5693 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20250812 13:58:09.994837 5709 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:09.995916 5693 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20250812 13:58:09.996228 5693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "63d8666af9b34b9499723813c60e1996"
format_stamp: "Formatted at 2025-08-12 13:58:09 on dist-test-slave-f6np"
I20250812 13:58:09.996567 5693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:10.071339 5693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:10.072818 5693 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:10.073268 5693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:10.076263 5693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:10.080442 5693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:10.080648 5693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:10.080874 5693 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:10.081025 5693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:10.231272 5693 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:41137
I20250812 13:58:10.231434 5821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:41137 every 8 connection(s)
I20250812 13:58:10.233778 5693 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 13:58:10.238703 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 5693
I20250812 13:58:10.239068 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 13:58:10.246127 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:0
--local_ip_for_outbound_sockets=127.4.158.2
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:34825
--builtin_ntp_servers=127.4.158.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250812 13:58:10.263092 5822 heartbeater.cc:344] Connected to a master server at 127.4.158.62:34825
I20250812 13:58:10.263641 5822 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:10.264966 5822 heartbeater.cc:507] Master 127.4.158.62:34825 requested a full tablet report, sending...
I20250812 13:58:10.267962 5634 ts_manager.cc:194] Registered new tserver with Master: 63d8666af9b34b9499723813c60e1996 (127.4.158.1:41137)
I20250812 13:58:10.270704 5634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:48671
W20250812 13:58:10.541605 5826 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250812 13:58:10.542321 5826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:10.542594 5826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:10.543046 5826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:10.574003 5826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:10.574806 5826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 13:58:10.608484 5826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:34825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:10.609740 5826 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:10.611308 5826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:10.622997 5832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:11.274813 5822 heartbeater.cc:499] Master 127.4.158.62:34825 was elected leader, sending a full tablet report...
W20250812 13:58:10.626675 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:10.623517 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:11.701577 5834 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 13:58:11.701637 5826 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:11.705482 5826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:11.708035 5826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:11.709460 5826 hybrid_clock.cc:648] HybridClock initialized: now 1755007091709430 us; error 46 us; skew 500 ppm
I20250812 13:58:11.710454 5826 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:11.716578 5826 webserver.cc:480] Webserver started at http://127.4.158.2:37865/ using document root <none> and password file <none>
I20250812 13:58:11.717456 5826 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:11.717633 5826 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:11.718079 5826 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:11.722395 5826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d73ec832e5e54e06becde10e535a4579"
format_stamp: "Formatted at 2025-08-12 13:58:11 on dist-test-slave-f6np"
I20250812 13:58:11.723431 5826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d73ec832e5e54e06becde10e535a4579"
format_stamp: "Formatted at 2025-08-12 13:58:11 on dist-test-slave-f6np"
I20250812 13:58:11.730578 5826 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250812 13:58:11.736001 5842 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:11.737016 5826 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250812 13:58:11.737314 5826 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d73ec832e5e54e06becde10e535a4579"
format_stamp: "Formatted at 2025-08-12 13:58:11 on dist-test-slave-f6np"
I20250812 13:58:11.737599 5826 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:11.795608 5826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:11.797012 5826 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:11.797446 5826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:11.799872 5826 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:11.803855 5826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:11.804049 5826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:11.804251 5826 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:11.804380 5826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:11.929980 5826 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:42597
I20250812 13:58:11.930109 5954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:42597 every 8 connection(s)
I20250812 13:58:11.932494 5826 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 13:58:11.933005 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 5826
I20250812 13:58:11.933367 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 13:58:11.939847 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:0
--local_ip_for_outbound_sockets=127.4.158.3
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:34825
--builtin_ntp_servers=127.4.158.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250812 13:58:11.955961 5955 heartbeater.cc:344] Connected to a master server at 127.4.158.62:34825
I20250812 13:58:11.956522 5955 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:11.957930 5955 heartbeater.cc:507] Master 127.4.158.62:34825 requested a full tablet report, sending...
I20250812 13:58:11.960404 5634 ts_manager.cc:194] Registered new tserver with Master: d73ec832e5e54e06becde10e535a4579 (127.4.158.2:42597)
I20250812 13:58:11.961709 5634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:36937
W20250812 13:58:12.239296 5959 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250812 13:58:12.239933 5959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:12.240168 5959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:12.240633 5959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:12.272838 5959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:12.273712 5959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 13:58:12.309315 5959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:34825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:12.310685 5959 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:12.312422 5959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:12.324244 5965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:12.965415 5955 heartbeater.cc:499] Master 127.4.158.62:34825 was elected leader, sending a full tablet report...
W20250812 13:58:12.324754 5966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:13.488878 5967 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1162 milliseconds
W20250812 13:58:13.489286 5968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:13.492352 5959 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.168s user 0.460s sys 0.697s
W20250812 13:58:13.492735 5959 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.168s user 0.461s sys 0.698s
I20250812 13:58:13.493001 5959 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:13.494213 5959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:13.496353 5959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:13.497689 5959 hybrid_clock.cc:648] HybridClock initialized: now 1755007093497659 us; error 44 us; skew 500 ppm
I20250812 13:58:13.498483 5959 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:13.505666 5959 webserver.cc:480] Webserver started at http://127.4.158.3:39855/ using document root <none> and password file <none>
I20250812 13:58:13.506683 5959 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:13.506891 5959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:13.507328 5959 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:13.511765 5959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fa6c9cdc1dd34858942f240f217094b0"
format_stamp: "Formatted at 2025-08-12 13:58:13 on dist-test-slave-f6np"
I20250812 13:58:13.512816 5959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fa6c9cdc1dd34858942f240f217094b0"
format_stamp: "Formatted at 2025-08-12 13:58:13 on dist-test-slave-f6np"
I20250812 13:58:13.520368 5959 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250812 13:58:13.526940 5975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:13.528066 5959 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250812 13:58:13.528419 5959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fa6c9cdc1dd34858942f240f217094b0"
format_stamp: "Formatted at 2025-08-12 13:58:13 on dist-test-slave-f6np"
I20250812 13:58:13.528734 5959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:13.603981 5959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:13.605427 5959 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:13.605836 5959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:13.608479 5959 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:13.612812 5959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:13.613040 5959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:13.613389 5959 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:13.613612 5959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:13.740044 5959 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:36499
I20250812 13:58:13.740130 6087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:36499 every 8 connection(s)
I20250812 13:58:13.742583 5959 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 13:58:13.745414 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 5959
I20250812 13:58:13.745893 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 13:58:13.767025 6088 heartbeater.cc:344] Connected to a master server at 127.4.158.62:34825
I20250812 13:58:13.767470 6088 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:13.768461 6088 heartbeater.cc:507] Master 127.4.158.62:34825 requested a full tablet report, sending...
I20250812 13:58:13.770541 5634 ts_manager.cc:194] Registered new tserver with Master: fa6c9cdc1dd34858942f240f217094b0 (127.4.158.3:36499)
I20250812 13:58:13.771723 5634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:45603
I20250812 13:58:13.779933 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:58:13.812944 5634 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55452:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250812 13:58:13.831211 5634 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 13:58:13.878885 5890 tablet_service.cc:1468] Processing CreateTablet for tablet 56aaa0e4b2724a3dba492341f9416cb3 (DEFAULT_TABLE table=TestTable [id=d94eec14cdb4442d98553a5c203241bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:13.881623 5890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56aaa0e4b2724a3dba492341f9416cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:13.884792 5757 tablet_service.cc:1468] Processing CreateTablet for tablet 56aaa0e4b2724a3dba492341f9416cb3 (DEFAULT_TABLE table=TestTable [id=d94eec14cdb4442d98553a5c203241bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:13.886530 5757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56aaa0e4b2724a3dba492341f9416cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:13.888123 6023 tablet_service.cc:1468] Processing CreateTablet for tablet 56aaa0e4b2724a3dba492341f9416cb3 (DEFAULT_TABLE table=TestTable [id=d94eec14cdb4442d98553a5c203241bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:13.889966 6023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56aaa0e4b2724a3dba492341f9416cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:13.908778 6107 tablet_bootstrap.cc:492] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996: Bootstrap starting.
I20250812 13:58:13.911758 6108 tablet_bootstrap.cc:492] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579: Bootstrap starting.
I20250812 13:58:13.915786 6109 tablet_bootstrap.cc:492] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0: Bootstrap starting.
I20250812 13:58:13.916831 6107 tablet_bootstrap.cc:654] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:13.919263 6107 log.cc:826] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:13.919571 6108 tablet_bootstrap.cc:654] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:13.922019 6108 log.cc:826] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:13.922744 6109 tablet_bootstrap.cc:654] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:13.924901 6109 log.cc:826] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:13.925040 6107 tablet_bootstrap.cc:492] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996: No bootstrap required, opened a new log
I20250812 13:58:13.925578 6107 ts_tablet_manager.cc:1397] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20250812 13:58:13.928771 6108 tablet_bootstrap.cc:492] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579: No bootstrap required, opened a new log
I20250812 13:58:13.929381 6108 ts_tablet_manager.cc:1397] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20250812 13:58:13.935765 6109 tablet_bootstrap.cc:492] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0: No bootstrap required, opened a new log
I20250812 13:58:13.936254 6109 ts_tablet_manager.cc:1397] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.007s
I20250812 13:58:13.952930 6107 raft_consensus.cc:357] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:13.953915 6107 raft_consensus.cc:738] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d8666af9b34b9499723813c60e1996, State: Initialized, Role: FOLLOWER
I20250812 13:58:13.954658 6107 consensus_queue.cc:260] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:13.957470 6108 raft_consensus.cc:357] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:13.958535 6108 raft_consensus.cc:738] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73ec832e5e54e06becde10e535a4579, State: Initialized, Role: FOLLOWER
I20250812 13:58:13.959340 6108 consensus_queue.cc:260] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:13.963102 6109 raft_consensus.cc:357] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:13.963984 6109 raft_consensus.cc:738] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6c9cdc1dd34858942f240f217094b0, State: Initialized, Role: FOLLOWER
I20250812 13:58:13.964926 6107 ts_tablet_manager.cc:1428] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996: Time spent starting tablet: real 0.039s user 0.028s sys 0.011s
I20250812 13:58:13.964680 6109 consensus_queue.cc:260] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:13.969818 6108 ts_tablet_manager.cc:1428] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579: Time spent starting tablet: real 0.040s user 0.030s sys 0.007s
I20250812 13:58:13.974987 6088 heartbeater.cc:499] Master 127.4.158.62:34825 was elected leader, sending a full tablet report...
I20250812 13:58:13.977838 6109 ts_tablet_manager.cc:1428] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0: Time spent starting tablet: real 0.041s user 0.029s sys 0.011s
W20250812 13:58:13.992154 5823 tablet.cc:2378] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 13:58:13.996997 6089 tablet.cc:2378] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:58:14.001673 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:58:14.005012 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 63d8666af9b34b9499723813c60e1996 to finish bootstrapping
I20250812 13:58:14.017704 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d73ec832e5e54e06becde10e535a4579 to finish bootstrapping
I20250812 13:58:14.027526 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fa6c9cdc1dd34858942f240f217094b0 to finish bootstrapping
I20250812 13:58:14.072131 5777 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "56aaa0e4b2724a3dba492341f9416cb3"
dest_uuid: "63d8666af9b34b9499723813c60e1996"
from {username='slave'} at 127.0.0.1:60578
I20250812 13:58:14.072697 5777 raft_consensus.cc:491] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250812 13:58:14.073004 5777 raft_consensus.cc:3058] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:14.077447 5777 raft_consensus.cc:513] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:14.079717 5777 leader_election.cc:290] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [CANDIDATE]: Term 1 election: Requested vote from peers d73ec832e5e54e06becde10e535a4579 (127.4.158.2:42597), fa6c9cdc1dd34858942f240f217094b0 (127.4.158.3:36499)
I20250812 13:58:14.089715 4728 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250812 13:58:14.091075 5910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56aaa0e4b2724a3dba492341f9416cb3" candidate_uuid: "63d8666af9b34b9499723813c60e1996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d73ec832e5e54e06becde10e535a4579"
I20250812 13:58:14.091665 5910 raft_consensus.cc:3058] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:14.091825 6043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56aaa0e4b2724a3dba492341f9416cb3" candidate_uuid: "63d8666af9b34b9499723813c60e1996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fa6c9cdc1dd34858942f240f217094b0"
I20250812 13:58:14.092396 6043 raft_consensus.cc:3058] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:14.096107 5910 raft_consensus.cc:2466] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d8666af9b34b9499723813c60e1996 in term 1.
I20250812 13:58:14.096817 6043 raft_consensus.cc:2466] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63d8666af9b34b9499723813c60e1996 in term 1.
I20250812 13:58:14.097257 5710 leader_election.cc:304] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63d8666af9b34b9499723813c60e1996, d73ec832e5e54e06becde10e535a4579; no voters:
I20250812 13:58:14.098142 6113 raft_consensus.cc:2802] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:14.099718 6113 raft_consensus.cc:695] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 1 LEADER]: Becoming Leader. State: Replica: 63d8666af9b34b9499723813c60e1996, State: Running, Role: LEADER
I20250812 13:58:14.100463 6113 consensus_queue.cc:237] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:14.110325 5633 catalog_manager.cc:5582] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 reported cstate change: term changed from 0 to 1, leader changed from <none> to 63d8666af9b34b9499723813c60e1996 (127.4.158.1). New cstate: current_term: 1 leader_uuid: "63d8666af9b34b9499723813c60e1996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } health_report { overall_health: UNKNOWN } } }
W20250812 13:58:14.189289 5956 tablet.cc:2378] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:58:14.195210 4728 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250812 13:58:14.400655 4728 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250812 13:58:14.513489 6113 consensus_queue.cc:1035] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 13:58:14.529821 6120 consensus_queue.cc:1035] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250812 13:58:16.279417 5777 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "56aaa0e4b2724a3dba492341f9416cb3"
dest_uuid: "63d8666af9b34b9499723813c60e1996"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60594
I20250812 13:58:16.280056 5777 raft_consensus.cc:604] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 1 LEADER]: Received request to transfer leadership
I20250812 13:58:16.460263 6142 raft_consensus.cc:991] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996: : Instructing follower d73ec832e5e54e06becde10e535a4579 to start an election
I20250812 13:58:16.460641 6153 raft_consensus.cc:1079] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 1 LEADER]: Signalling peer d73ec832e5e54e06becde10e535a4579 to start an election
I20250812 13:58:16.462077 5910 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "56aaa0e4b2724a3dba492341f9416cb3"
dest_uuid: "d73ec832e5e54e06becde10e535a4579"
from {username='slave'} at 127.4.158.1:35413
I20250812 13:58:16.462594 5910 raft_consensus.cc:491] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250812 13:58:16.462853 5910 raft_consensus.cc:3058] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 1 FOLLOWER]: Advancing to term 2
I20250812 13:58:16.466892 5910 raft_consensus.cc:513] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:16.468986 5910 leader_election.cc:290] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [CANDIDATE]: Term 2 election: Requested vote from peers 63d8666af9b34b9499723813c60e1996 (127.4.158.1:41137), fa6c9cdc1dd34858942f240f217094b0 (127.4.158.3:36499)
I20250812 13:58:16.479791 5777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56aaa0e4b2724a3dba492341f9416cb3" candidate_uuid: "d73ec832e5e54e06becde10e535a4579" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "63d8666af9b34b9499723813c60e1996"
I20250812 13:58:16.480342 5777 raft_consensus.cc:3053] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 1 LEADER]: Stepping down as leader of term 1
I20250812 13:58:16.480464 6043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56aaa0e4b2724a3dba492341f9416cb3" candidate_uuid: "d73ec832e5e54e06becde10e535a4579" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa6c9cdc1dd34858942f240f217094b0"
I20250812 13:58:16.480656 5777 raft_consensus.cc:738] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 63d8666af9b34b9499723813c60e1996, State: Running, Role: LEADER
I20250812 13:58:16.480995 6043 raft_consensus.cc:3058] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0 [term 1 FOLLOWER]: Advancing to term 2
I20250812 13:58:16.481284 5777 consensus_queue.cc:260] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:16.482486 5777 raft_consensus.cc:3058] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 1 FOLLOWER]: Advancing to term 2
I20250812 13:58:16.487251 6043 raft_consensus.cc:2466] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73ec832e5e54e06becde10e535a4579 in term 2.
I20250812 13:58:16.488550 5777 raft_consensus.cc:2466] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d73ec832e5e54e06becde10e535a4579 in term 2.
I20250812 13:58:16.488510 5845 leader_election.cc:304] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d73ec832e5e54e06becde10e535a4579, fa6c9cdc1dd34858942f240f217094b0; no voters:
I20250812 13:58:16.490516 6157 raft_consensus.cc:2802] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 13:58:16.491807 6157 raft_consensus.cc:695] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [term 2 LEADER]: Becoming Leader. State: Replica: d73ec832e5e54e06becde10e535a4579, State: Running, Role: LEADER
I20250812 13:58:16.492632 6157 consensus_queue.cc:237] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } }
I20250812 13:58:16.499990 5632 catalog_manager.cc:5582] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 reported cstate change: term changed from 1 to 2, leader changed from 63d8666af9b34b9499723813c60e1996 (127.4.158.1) to d73ec832e5e54e06becde10e535a4579 (127.4.158.2). New cstate: current_term: 2 leader_uuid: "d73ec832e5e54e06becde10e535a4579" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d73ec832e5e54e06becde10e535a4579" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 } health_report { overall_health: UNKNOWN } } }
I20250812 13:58:17.053153 5777 raft_consensus.cc:1273] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 2 FOLLOWER]: Refusing update from remote peer d73ec832e5e54e06becde10e535a4579: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250812 13:58:17.054677 6165 consensus_queue.cc:1035] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63d8666af9b34b9499723813c60e1996" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 41137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250812 13:58:17.065655 6043 raft_consensus.cc:1273] T 56aaa0e4b2724a3dba492341f9416cb3 P fa6c9cdc1dd34858942f240f217094b0 [term 2 FOLLOWER]: Refusing update from remote peer d73ec832e5e54e06becde10e535a4579: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250812 13:58:17.067384 6168 consensus_queue.cc:1035] T 56aaa0e4b2724a3dba492341f9416cb3 P d73ec832e5e54e06becde10e535a4579 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6c9cdc1dd34858942f240f217094b0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250812 13:58:18.882786 5777 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "56aaa0e4b2724a3dba492341f9416cb3"
dest_uuid: "63d8666af9b34b9499723813c60e1996"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43510
I20250812 13:58:18.883574 5777 raft_consensus.cc:604] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 2 FOLLOWER]: Received request to transfer leadership
I20250812 13:58:18.883965 5777 raft_consensus.cc:612] T 56aaa0e4b2724a3dba492341f9416cb3 P 63d8666af9b34b9499723813c60e1996 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250812 13:58:19.920672 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 5693
I20250812 13:58:19.940632 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 5826
I20250812 13:58:19.960649 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 5959
I20250812 13:58:19.983161 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 5600
2025-08-12T13:58:19Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14109 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250812 13:58:20.035984 4728 test_util.cc:276] Using random seed: 1858511724
I20250812 13:58:20.040135 4728 ts_itest-base.cc:115] Starting cluster with:
I20250812 13:58:20.040306 4728 ts_itest-base.cc:116] --------------
I20250812 13:58:20.040439 4728 ts_itest-base.cc:117] 3 tablet servers
I20250812 13:58:20.040549 4728 ts_itest-base.cc:118] 3 replicas per TS
I20250812 13:58:20.040657 4728 ts_itest-base.cc:119] --------------
2025-08-12T13:58:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T13:58:20Z Disabled control of system clock
I20250812 13:58:20.073189 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:45733
--webserver_interface=127.4.158.62
--webserver_port=0
--builtin_ntp_servers=127.4.158.20:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:45733 with env {}
W20250812 13:58:20.365239 6203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:20.365803 6203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:20.366307 6203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:20.398016 6203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:58:20.398330 6203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:20.398535 6203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:58:20.398730 6203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:58:20.433740 6203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:45733
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:45733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:20.435017 6203 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:20.436605 6203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:20.447275 6209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:20.447716 6210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:20.453425 6212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:21.541715 6211 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 13:58:21.541743 6203 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:21.545737 6203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:21.548653 6203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:21.550071 6203 hybrid_clock.cc:648] HybridClock initialized: now 1755007101550030 us; error 55 us; skew 500 ppm
I20250812 13:58:21.550928 6203 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:21.557317 6203 webserver.cc:480] Webserver started at http://127.4.158.62:45183/ using document root <none> and password file <none>
I20250812 13:58:21.558262 6203 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:21.558480 6203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:21.558938 6203 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:21.563596 6203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f99fdbe9fc0a41048c0fd13e189c9c77"
format_stamp: "Formatted at 2025-08-12 13:58:21 on dist-test-slave-f6np"
I20250812 13:58:21.564669 6203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f99fdbe9fc0a41048c0fd13e189c9c77"
format_stamp: "Formatted at 2025-08-12 13:58:21 on dist-test-slave-f6np"
I20250812 13:58:21.571676 6203 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250812 13:58:21.576965 6219 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:21.577989 6203 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250812 13:58:21.578300 6203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f99fdbe9fc0a41048c0fd13e189c9c77"
format_stamp: "Formatted at 2025-08-12 13:58:21 on dist-test-slave-f6np"
I20250812 13:58:21.578629 6203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:21.630496 6203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:21.631997 6203 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:21.632429 6203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:21.698846 6203 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:45733
I20250812 13:58:21.698916 6270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:45733 every 8 connection(s)
I20250812 13:58:21.701462 6203 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:58:21.703891 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 6203
I20250812 13:58:21.704360 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 13:58:21.706804 6271 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:21.729933 6271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77: Bootstrap starting.
I20250812 13:58:21.735219 6271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:21.736829 6271 log.cc:826] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:21.741062 6271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77: No bootstrap required, opened a new log
I20250812 13:58:21.757586 6271 raft_consensus.cc:357] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99fdbe9fc0a41048c0fd13e189c9c77" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 45733 } }
I20250812 13:58:21.758266 6271 raft_consensus.cc:383] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:21.758491 6271 raft_consensus.cc:738] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99fdbe9fc0a41048c0fd13e189c9c77, State: Initialized, Role: FOLLOWER
I20250812 13:58:21.759171 6271 consensus_queue.cc:260] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99fdbe9fc0a41048c0fd13e189c9c77" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 45733 } }
I20250812 13:58:21.759656 6271 raft_consensus.cc:397] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:58:21.759902 6271 raft_consensus.cc:491] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:58:21.760202 6271 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:21.764640 6271 raft_consensus.cc:513] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99fdbe9fc0a41048c0fd13e189c9c77" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 45733 } }
I20250812 13:58:21.765270 6271 leader_election.cc:304] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f99fdbe9fc0a41048c0fd13e189c9c77; no voters:
I20250812 13:58:21.766849 6271 leader_election.cc:290] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:58:21.767472 6276 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:21.769340 6276 raft_consensus.cc:695] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [term 1 LEADER]: Becoming Leader. State: Replica: f99fdbe9fc0a41048c0fd13e189c9c77, State: Running, Role: LEADER
I20250812 13:58:21.770090 6276 consensus_queue.cc:237] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99fdbe9fc0a41048c0fd13e189c9c77" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 45733 } }
I20250812 13:58:21.771071 6271 sys_catalog.cc:564] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:58:21.779055 6277 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f99fdbe9fc0a41048c0fd13e189c9c77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99fdbe9fc0a41048c0fd13e189c9c77" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 45733 } } }
I20250812 13:58:21.779206 6278 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f99fdbe9fc0a41048c0fd13e189c9c77. Latest consensus state: current_term: 1 leader_uuid: "f99fdbe9fc0a41048c0fd13e189c9c77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99fdbe9fc0a41048c0fd13e189c9c77" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 45733 } } }
I20250812 13:58:21.779778 6278 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:21.779778 6277 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:21.783128 6284 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:58:21.793630 6284 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:58:21.809487 6284 catalog_manager.cc:1349] Generated new cluster ID: 741e0ac485594e6ba23cd796f37ea240
I20250812 13:58:21.809743 6284 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:58:21.825222 6284 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 13:58:21.826696 6284 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:58:21.841238 6284 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f99fdbe9fc0a41048c0fd13e189c9c77: Generated new TSK 0
I20250812 13:58:21.842173 6284 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 13:58:21.866870 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:0
--local_ip_for_outbound_sockets=127.4.158.1
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:45733
--builtin_ntp_servers=127.4.158.20:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 13:58:22.196827 6295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:22.197328 6295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:22.197817 6295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:22.229540 6295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:22.230424 6295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 13:58:22.266023 6295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:45733
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:22.267340 6295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:22.268918 6295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:22.281528 6301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:23.683038 6300 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 6295
W20250812 13:58:23.838640 6300 kernel_stack_watchdog.cc:198] Thread 6295 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 13:58:23.839207 6295 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.558s user 0.564s sys 0.950s
W20250812 13:58:23.839619 6295 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.558s user 0.564s sys 0.951s
W20250812 13:58:22.282892 6302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:23.840746 6303 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1558 milliseconds
I20250812 13:58:23.841774 6295 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250812 13:58:23.841828 6304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:23.845074 6295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:23.847258 6295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:23.848620 6295 hybrid_clock.cc:648] HybridClock initialized: now 1755007103848577 us; error 53 us; skew 500 ppm
I20250812 13:58:23.849399 6295 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:23.855415 6295 webserver.cc:480] Webserver started at http://127.4.158.1:45403/ using document root <none> and password file <none>
I20250812 13:58:23.856324 6295 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:23.856534 6295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:23.856962 6295 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:23.861356 6295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9e274bdcb0144f7282b3b4116571a282"
format_stamp: "Formatted at 2025-08-12 13:58:23 on dist-test-slave-f6np"
I20250812 13:58:23.862530 6295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9e274bdcb0144f7282b3b4116571a282"
format_stamp: "Formatted at 2025-08-12 13:58:23 on dist-test-slave-f6np"
I20250812 13:58:23.869786 6295 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250812 13:58:23.875726 6311 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:23.876899 6295 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250812 13:58:23.877223 6295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9e274bdcb0144f7282b3b4116571a282"
format_stamp: "Formatted at 2025-08-12 13:58:23 on dist-test-slave-f6np"
I20250812 13:58:23.877609 6295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:23.944253 6295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:23.945714 6295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:23.946166 6295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:23.948730 6295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:23.952859 6295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:23.953063 6295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:23.953308 6295 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:23.953459 6295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:24.103665 6295 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:46115
I20250812 13:58:24.103768 6423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:46115 every 8 connection(s)
I20250812 13:58:24.106375 6295 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 13:58:24.107230 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 6295
I20250812 13:58:24.107694 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 13:58:24.115564 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:0
--local_ip_for_outbound_sockets=127.4.158.2
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:45733
--builtin_ntp_servers=127.4.158.20:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:58:24.142865 6424 heartbeater.cc:344] Connected to a master server at 127.4.158.62:45733
I20250812 13:58:24.143330 6424 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:24.144549 6424 heartbeater.cc:507] Master 127.4.158.62:45733 requested a full tablet report, sending...
I20250812 13:58:24.147055 6236 ts_manager.cc:194] Registered new tserver with Master: 9e274bdcb0144f7282b3b4116571a282 (127.4.158.1:46115)
I20250812 13:58:24.149067 6236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:52273
W20250812 13:58:24.432828 6428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:24.433339 6428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:24.433823 6428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:24.464835 6428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:24.465692 6428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 13:58:24.500370 6428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:45733
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:24.501689 6428 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:24.503268 6428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:24.515096 6434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:25.152489 6424 heartbeater.cc:499] Master 127.4.158.62:45733 was elected leader, sending a full tablet report...
W20250812 13:58:25.917676 6433 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 6428
W20250812 13:58:25.963373 6428 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.448s user 0.566s sys 0.877s
W20250812 13:58:25.963738 6428 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.449s user 0.566s sys 0.877s
W20250812 13:58:24.515573 6435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:25.965543 6437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:25.968202 6436 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1448 milliseconds
I20250812 13:58:25.968266 6428 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:25.969440 6428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:25.971429 6428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:25.972748 6428 hybrid_clock.cc:648] HybridClock initialized: now 1755007105972711 us; error 42 us; skew 500 ppm
I20250812 13:58:25.973500 6428 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:25.979034 6428 webserver.cc:480] Webserver started at http://127.4.158.2:35251/ using document root <none> and password file <none>
I20250812 13:58:25.979938 6428 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:25.980139 6428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:25.980568 6428 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:25.985234 6428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "28c7474fb1c14690bcec9b90c8c13d47"
format_stamp: "Formatted at 2025-08-12 13:58:25 on dist-test-slave-f6np"
I20250812 13:58:25.986351 6428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "28c7474fb1c14690bcec9b90c8c13d47"
format_stamp: "Formatted at 2025-08-12 13:58:25 on dist-test-slave-f6np"
I20250812 13:58:25.993053 6428 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250812 13:58:25.998386 6444 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:25.999369 6428 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250812 13:58:25.999672 6428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "28c7474fb1c14690bcec9b90c8c13d47"
format_stamp: "Formatted at 2025-08-12 13:58:25 on dist-test-slave-f6np"
I20250812 13:58:25.999991 6428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:26.042367 6428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:26.043881 6428 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:26.044292 6428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:26.046680 6428 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:26.051031 6428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:26.051251 6428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:26.051482 6428 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:26.051635 6428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:26.179515 6428 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:43605
I20250812 13:58:26.179627 6556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:43605 every 8 connection(s)
I20250812 13:58:26.182065 6428 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 13:58:26.186491 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 6428
I20250812 13:58:26.186829 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 13:58:26.193045 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:0
--local_ip_for_outbound_sockets=127.4.158.3
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:45733
--builtin_ntp_servers=127.4.158.20:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:58:26.205466 6557 heartbeater.cc:344] Connected to a master server at 127.4.158.62:45733
I20250812 13:58:26.205981 6557 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:26.207051 6557 heartbeater.cc:507] Master 127.4.158.62:45733 requested a full tablet report, sending...
I20250812 13:58:26.209486 6236 ts_manager.cc:194] Registered new tserver with Master: 28c7474fb1c14690bcec9b90c8c13d47 (127.4.158.2:43605)
I20250812 13:58:26.211231 6236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:47951
W20250812 13:58:26.485482 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:26.485965 6561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:26.486450 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:26.517326 6561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:26.518151 6561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 13:58:26.552103 6561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:45733
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:26.553323 6561 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:26.554924 6561 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:26.566860 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:27.214607 6557 heartbeater.cc:499] Master 127.4.158.62:45733 was elected leader, sending a full tablet report...
W20250812 13:58:26.567296 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:27.969280 6566 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 6561
W20250812 13:58:28.060719 6561 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.494s user 0.002s sys 0.000s
W20250812 13:58:28.061048 6561 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.494s user 0.002s sys 0.000s
W20250812 13:58:28.062772 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:28.064508 6569 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1492 milliseconds
I20250812 13:58:28.064529 6561 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:28.065667 6561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:28.068048 6561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:28.069448 6561 hybrid_clock.cc:648] HybridClock initialized: now 1755007108069419 us; error 35 us; skew 500 ppm
I20250812 13:58:28.070233 6561 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:28.075853 6561 webserver.cc:480] Webserver started at http://127.4.158.3:35425/ using document root <none> and password file <none>
I20250812 13:58:28.076731 6561 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:28.076933 6561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:28.077351 6561 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:28.081602 6561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8"
format_stamp: "Formatted at 2025-08-12 13:58:28 on dist-test-slave-f6np"
I20250812 13:58:28.082695 6561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8"
format_stamp: "Formatted at 2025-08-12 13:58:28 on dist-test-slave-f6np"
I20250812 13:58:28.089419 6561 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250812 13:58:28.094796 6578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:28.095778 6561 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250812 13:58:28.096073 6561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8"
format_stamp: "Formatted at 2025-08-12 13:58:28 on dist-test-slave-f6np"
I20250812 13:58:28.096385 6561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:28.154677 6561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:28.156091 6561 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:28.156524 6561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:28.159049 6561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:28.164630 6561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:28.164906 6561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:28.165216 6561 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:28.165503 6561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250812 13:58:28.293581 6561 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:38463
I20250812 13:58:28.293711 6690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:38463 every 8 connection(s)
I20250812 13:58:28.296082 6561 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 13:58:28.296977 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 6561
I20250812 13:58:28.297508 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 13:58:28.317121 6691 heartbeater.cc:344] Connected to a master server at 127.4.158.62:45733
I20250812 13:58:28.317548 6691 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:28.318624 6691 heartbeater.cc:507] Master 127.4.158.62:45733 requested a full tablet report, sending...
I20250812 13:58:28.320633 6236 ts_manager.cc:194] Registered new tserver with Master: dcefdc411d0b4c7bb0d45aa8c3a880a8 (127.4.158.3:38463)
I20250812 13:58:28.321786 6236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:50567
I20250812 13:58:28.332058 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:58:28.362946 6236 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250812 13:58:28.380980 6236 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 13:58:28.437674 6626 tablet_service.cc:1468] Processing CreateTablet for tablet c63f4c160e4c4f6c83fb5de44cb22c54 (DEFAULT_TABLE table=TestTable [id=d65bd4eae1674a08ab976ffbc1d24f37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:28.440217 6626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c63f4c160e4c4f6c83fb5de44cb22c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:28.442205 6492 tablet_service.cc:1468] Processing CreateTablet for tablet c63f4c160e4c4f6c83fb5de44cb22c54 (DEFAULT_TABLE table=TestTable [id=d65bd4eae1674a08ab976ffbc1d24f37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:28.444006 6492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c63f4c160e4c4f6c83fb5de44cb22c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:28.444211 6359 tablet_service.cc:1468] Processing CreateTablet for tablet c63f4c160e4c4f6c83fb5de44cb22c54 (DEFAULT_TABLE table=TestTable [id=d65bd4eae1674a08ab976ffbc1d24f37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:28.445621 6359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c63f4c160e4c4f6c83fb5de44cb22c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:28.466140 6710 tablet_bootstrap.cc:492] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282: Bootstrap starting.
I20250812 13:58:28.468391 6711 tablet_bootstrap.cc:492] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Bootstrap starting.
I20250812 13:58:28.471624 6712 tablet_bootstrap.cc:492] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47: Bootstrap starting.
I20250812 13:58:28.473958 6710 tablet_bootstrap.cc:654] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:28.474647 6711 tablet_bootstrap.cc:654] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:28.476310 6711 log.cc:826] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:28.476284 6710 log.cc:826] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:28.479763 6712 tablet_bootstrap.cc:654] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:28.482177 6712 log.cc:826] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:28.484268 6711 tablet_bootstrap.cc:492] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8: No bootstrap required, opened a new log
I20250812 13:58:28.484566 6710 tablet_bootstrap.cc:492] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282: No bootstrap required, opened a new log
I20250812 13:58:28.484760 6711 ts_tablet_manager.cc:1397] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20250812 13:58:28.485018 6710 ts_tablet_manager.cc:1397] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20250812 13:58:28.486812 6712 tablet_bootstrap.cc:492] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47: No bootstrap required, opened a new log
I20250812 13:58:28.487178 6712 ts_tablet_manager.cc:1397] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250812 13:58:28.504410 6712 raft_consensus.cc:357] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.505079 6712 raft_consensus.cc:383] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:28.505331 6712 raft_consensus.cc:738] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c7474fb1c14690bcec9b90c8c13d47, State: Initialized, Role: FOLLOWER
I20250812 13:58:28.506090 6712 consensus_queue.cc:260] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.509869 6712 ts_tablet_manager.cc:1428] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47: Time spent starting tablet: real 0.022s user 0.018s sys 0.004s
I20250812 13:58:28.510465 6710 raft_consensus.cc:357] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.510448 6711 raft_consensus.cc:357] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.511380 6710 raft_consensus.cc:383] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:28.511447 6711 raft_consensus.cc:383] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:28.511731 6710 raft_consensus.cc:738] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e274bdcb0144f7282b3b4116571a282, State: Initialized, Role: FOLLOWER
I20250812 13:58:28.511744 6711 raft_consensus.cc:738] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcefdc411d0b4c7bb0d45aa8c3a880a8, State: Initialized, Role: FOLLOWER
I20250812 13:58:28.512591 6710 consensus_queue.cc:260] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.512656 6711 consensus_queue.cc:260] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.515801 6691 heartbeater.cc:499] Master 127.4.158.62:45733 was elected leader, sending a full tablet report...
I20250812 13:58:28.517359 6711 ts_tablet_manager.cc:1428] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Time spent starting tablet: real 0.032s user 0.014s sys 0.017s
I20250812 13:58:28.519275 6710 ts_tablet_manager.cc:1428] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282: Time spent starting tablet: real 0.034s user 0.030s sys 0.004s
W20250812 13:58:28.550563 6692 tablet.cc:2378] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:58:28.554714 6716 raft_consensus.cc:491] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:58:28.555243 6716 raft_consensus.cc:513] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.557906 6716 leader_election.cc:290] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcefdc411d0b4c7bb0d45aa8c3a880a8 (127.4.158.3:38463), 9e274bdcb0144f7282b3b4116571a282 (127.4.158.1:46115)
I20250812 13:58:28.571159 6646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63f4c160e4c4f6c83fb5de44cb22c54" candidate_uuid: "28c7474fb1c14690bcec9b90c8c13d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" is_pre_election: true
I20250812 13:58:28.571308 6379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63f4c160e4c4f6c83fb5de44cb22c54" candidate_uuid: "28c7474fb1c14690bcec9b90c8c13d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e274bdcb0144f7282b3b4116571a282" is_pre_election: true
I20250812 13:58:28.571913 6646 raft_consensus.cc:2466] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28c7474fb1c14690bcec9b90c8c13d47 in term 0.
I20250812 13:58:28.572067 6379 raft_consensus.cc:2466] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28c7474fb1c14690bcec9b90c8c13d47 in term 0.
I20250812 13:58:28.573118 6447 leader_election.cc:304] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28c7474fb1c14690bcec9b90c8c13d47, dcefdc411d0b4c7bb0d45aa8c3a880a8; no voters:
I20250812 13:58:28.573779 6716 raft_consensus.cc:2802] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 13:58:28.574115 6716 raft_consensus.cc:491] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:58:28.574323 6716 raft_consensus.cc:3058] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:28.579212 6716 raft_consensus.cc:513] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.580783 6716 leader_election.cc:290] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [CANDIDATE]: Term 1 election: Requested vote from peers dcefdc411d0b4c7bb0d45aa8c3a880a8 (127.4.158.3:38463), 9e274bdcb0144f7282b3b4116571a282 (127.4.158.1:46115)
I20250812 13:58:28.581619 6379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63f4c160e4c4f6c83fb5de44cb22c54" candidate_uuid: "28c7474fb1c14690bcec9b90c8c13d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e274bdcb0144f7282b3b4116571a282"
I20250812 13:58:28.581626 6646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c63f4c160e4c4f6c83fb5de44cb22c54" candidate_uuid: "28c7474fb1c14690bcec9b90c8c13d47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8"
I20250812 13:58:28.582021 6379 raft_consensus.cc:3058] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:28.582028 6646 raft_consensus.cc:3058] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:28.586333 6379 raft_consensus.cc:2466] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28c7474fb1c14690bcec9b90c8c13d47 in term 1.
I20250812 13:58:28.586392 6646 raft_consensus.cc:2466] T c63f4c160e4c4f6c83fb5de44cb22c54 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28c7474fb1c14690bcec9b90c8c13d47 in term 1.
I20250812 13:58:28.587220 6445 leader_election.cc:304] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28c7474fb1c14690bcec9b90c8c13d47, 9e274bdcb0144f7282b3b4116571a282; no voters:
I20250812 13:58:28.588044 6716 raft_consensus.cc:2802] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:28.589926 6716 raft_consensus.cc:695] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [term 1 LEADER]: Becoming Leader. State: Replica: 28c7474fb1c14690bcec9b90c8c13d47, State: Running, Role: LEADER
I20250812 13:58:28.590740 6716 consensus_queue.cc:237] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } }
I20250812 13:58:28.600725 6234 catalog_manager.cc:5582] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28c7474fb1c14690bcec9b90c8c13d47 (127.4.158.2). New cstate: current_term: 1 leader_uuid: "28c7474fb1c14690bcec9b90c8c13d47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } health_report { overall_health: UNKNOWN } } }
W20250812 13:58:28.617117 6425 tablet.cc:2378] T c63f4c160e4c4f6c83fb5de44cb22c54 P 9e274bdcb0144f7282b3b4116571a282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:58:28.636402 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:58:28.639743 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9e274bdcb0144f7282b3b4116571a282 to finish bootstrapping
I20250812 13:58:28.651569 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 28c7474fb1c14690bcec9b90c8c13d47 to finish bootstrapping
I20250812 13:58:28.661276 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver dcefdc411d0b4c7bb0d45aa8c3a880a8 to finish bootstrapping
I20250812 13:58:28.673739 6234 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47922:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250812 13:58:28.675236 6234 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250812 13:58:28.688705 6558 tablet.cc:2378] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:58:28.691906 6359 tablet_service.cc:1468] Processing CreateTablet for tablet fb793124e301418383856d7829b3d063 (DEFAULT_TABLE table=TestAnotherTable [id=6c6c585308844844a9be3fd384363540]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250812 13:58:28.692569 6492 tablet_service.cc:1468] Processing CreateTablet for tablet fb793124e301418383856d7829b3d063 (DEFAULT_TABLE table=TestAnotherTable [id=6c6c585308844844a9be3fd384363540]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250812 13:58:28.693001 6359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb793124e301418383856d7829b3d063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:28.693059 6626 tablet_service.cc:1468] Processing CreateTablet for tablet fb793124e301418383856d7829b3d063 (DEFAULT_TABLE table=TestAnotherTable [id=6c6c585308844844a9be3fd384363540]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250812 13:58:28.693581 6492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb793124e301418383856d7829b3d063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:28.693826 6626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb793124e301418383856d7829b3d063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:28.707016 6710 tablet_bootstrap.cc:492] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282: Bootstrap starting.
I20250812 13:58:28.707113 6711 tablet_bootstrap.cc:492] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Bootstrap starting.
I20250812 13:58:28.712869 6711 tablet_bootstrap.cc:654] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:28.713385 6710 tablet_bootstrap.cc:654] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:28.717032 6712 tablet_bootstrap.cc:492] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47: Bootstrap starting.
I20250812 13:58:28.718389 6711 tablet_bootstrap.cc:492] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8: No bootstrap required, opened a new log
I20250812 13:58:28.718845 6711 ts_tablet_manager.cc:1397] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20250812 13:58:28.721408 6711 raft_consensus.cc:357] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.722167 6711 raft_consensus.cc:383] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:28.722486 6711 raft_consensus.cc:738] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcefdc411d0b4c7bb0d45aa8c3a880a8, State: Initialized, Role: FOLLOWER
I20250812 13:58:28.722746 6712 tablet_bootstrap.cc:654] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:28.723325 6711 consensus_queue.cc:260] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.726346 6710 tablet_bootstrap.cc:492] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282: No bootstrap required, opened a new log
I20250812 13:58:28.726784 6710 ts_tablet_manager.cc:1397] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.000s
I20250812 13:58:28.729314 6711 ts_tablet_manager.cc:1428] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250812 13:58:28.729460 6710 raft_consensus.cc:357] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.730165 6710 raft_consensus.cc:383] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:28.730444 6710 raft_consensus.cc:738] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e274bdcb0144f7282b3b4116571a282, State: Initialized, Role: FOLLOWER
I20250812 13:58:28.731118 6710 consensus_queue.cc:260] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.733139 6710 ts_tablet_manager.cc:1428] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250812 13:58:28.737514 6712 tablet_bootstrap.cc:492] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47: No bootstrap required, opened a new log
I20250812 13:58:28.738030 6712 ts_tablet_manager.cc:1397] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.007s
I20250812 13:58:28.740581 6712 raft_consensus.cc:357] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.741240 6712 raft_consensus.cc:383] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:28.741532 6712 raft_consensus.cc:738] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c7474fb1c14690bcec9b90c8c13d47, State: Initialized, Role: FOLLOWER
I20250812 13:58:28.742174 6712 consensus_queue.cc:260] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.744181 6712 ts_tablet_manager.cc:1428] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250812 13:58:28.796173 6718 raft_consensus.cc:491] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:58:28.796669 6718 raft_consensus.cc:513] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.799119 6718 leader_election.cc:290] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e274bdcb0144f7282b3b4116571a282 (127.4.158.1:46115), 28c7474fb1c14690bcec9b90c8c13d47 (127.4.158.2:43605)
I20250812 13:58:28.807070 6379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb793124e301418383856d7829b3d063" candidate_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e274bdcb0144f7282b3b4116571a282" is_pre_election: true
I20250812 13:58:28.807540 6379 raft_consensus.cc:2466] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcefdc411d0b4c7bb0d45aa8c3a880a8 in term 0.
I20250812 13:58:28.808492 6579 leader_election.cc:304] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e274bdcb0144f7282b3b4116571a282, dcefdc411d0b4c7bb0d45aa8c3a880a8; no voters:
I20250812 13:58:28.809161 6718 raft_consensus.cc:2802] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 13:58:28.809564 6718 raft_consensus.cc:491] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:58:28.809872 6718 raft_consensus.cc:3058] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:28.810322 6512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb793124e301418383856d7829b3d063" candidate_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c7474fb1c14690bcec9b90c8c13d47" is_pre_election: true
I20250812 13:58:28.811098 6512 raft_consensus.cc:2466] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcefdc411d0b4c7bb0d45aa8c3a880a8 in term 0.
I20250812 13:58:28.814975 6718 raft_consensus.cc:513] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.816350 6718 leader_election.cc:290] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [CANDIDATE]: Term 1 election: Requested vote from peers 9e274bdcb0144f7282b3b4116571a282 (127.4.158.1:46115), 28c7474fb1c14690bcec9b90c8c13d47 (127.4.158.2:43605)
I20250812 13:58:28.817095 6379 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb793124e301418383856d7829b3d063" candidate_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e274bdcb0144f7282b3b4116571a282"
I20250812 13:58:28.817225 6512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb793124e301418383856d7829b3d063" candidate_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28c7474fb1c14690bcec9b90c8c13d47"
I20250812 13:58:28.817625 6512 raft_consensus.cc:3058] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:28.817605 6379 raft_consensus.cc:3058] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:28.821516 6379 raft_consensus.cc:2466] T fb793124e301418383856d7829b3d063 P 9e274bdcb0144f7282b3b4116571a282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcefdc411d0b4c7bb0d45aa8c3a880a8 in term 1.
I20250812 13:58:28.821810 6512 raft_consensus.cc:2466] T fb793124e301418383856d7829b3d063 P 28c7474fb1c14690bcec9b90c8c13d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcefdc411d0b4c7bb0d45aa8c3a880a8 in term 1.
I20250812 13:58:28.822340 6579 leader_election.cc:304] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e274bdcb0144f7282b3b4116571a282, dcefdc411d0b4c7bb0d45aa8c3a880a8; no voters:
I20250812 13:58:28.822930 6718 raft_consensus.cc:2802] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:28.824894 6718 raft_consensus.cc:695] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [term 1 LEADER]: Becoming Leader. State: Replica: dcefdc411d0b4c7bb0d45aa8c3a880a8, State: Running, Role: LEADER
I20250812 13:58:28.825680 6718 consensus_queue.cc:237] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } }
I20250812 13:58:28.836369 6234 catalog_manager.cc:5582] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcefdc411d0b4c7bb0d45aa8c3a880a8 (127.4.158.3). New cstate: current_term: 1 leader_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 } health_report { overall_health: HEALTHY } } }
I20250812 13:58:29.025807 6716 consensus_queue.cc:1035] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 13:58:29.046975 6734 consensus_queue.cc:1035] T c63f4c160e4c4f6c83fb5de44cb22c54 P 28c7474fb1c14690bcec9b90c8c13d47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcefdc411d0b4c7bb0d45aa8c3a880a8" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250812 13:58:29.145064 6730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:29.145637 6730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:29.176146 6730 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250812 13:58:29.386257 6718 consensus_queue.cc:1035] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e274bdcb0144f7282b3b4116571a282" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250812 13:58:29.426748 6748 consensus_queue.cc:1035] T fb793124e301418383856d7829b3d063 P dcefdc411d0b4c7bb0d45aa8c3a880a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28c7474fb1c14690bcec9b90c8c13d47" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250812 13:58:30.383602 6730 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.170s user 0.499s sys 0.670s
W20250812 13:58:30.383992 6730 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.171s user 0.499s sys 0.670s
W20250812 13:58:31.759795 6755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:31.760380 6755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:31.791608 6755 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250812 13:58:33.017094 6755 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.187s user 0.478s sys 0.703s
W20250812 13:58:33.017477 6755 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.188s user 0.478s sys 0.703s
W20250812 13:58:34.398208 6770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:34.398774 6770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:34.431968 6770 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250812 13:58:35.616520 6770 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.147s user 0.454s sys 0.690s
W20250812 13:58:35.616909 6770 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.147s user 0.454s sys 0.690s
W20250812 13:58:36.988765 6788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:36.989315 6788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:37.020502 6788 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250812 13:58:38.196156 6788 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.138s user 0.464s sys 0.671s
W20250812 13:58:38.196523 6788 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.139s user 0.464s sys 0.671s
I20250812 13:58:39.267208 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 6295
I20250812 13:58:39.288378 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 6428
I20250812 13:58:39.308748 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 6561
I20250812 13:58:39.328804 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 6203
2025-08-12T13:58:39Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19344 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250812 13:58:39.380756 4728 test_util.cc:276] Using random seed: 1877856505
I20250812 13:58:39.384770 4728 ts_itest-base.cc:115] Starting cluster with:
I20250812 13:58:39.384927 4728 ts_itest-base.cc:116] --------------
I20250812 13:58:39.385088 4728 ts_itest-base.cc:117] 3 tablet servers
I20250812 13:58:39.385221 4728 ts_itest-base.cc:118] 3 replicas per TS
I20250812 13:58:39.385351 4728 ts_itest-base.cc:119] --------------
2025-08-12T13:58:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T13:58:39Z Disabled control of system clock
I20250812 13:58:39.417620 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:46869
--webserver_interface=127.4.158.62
--webserver_port=0
--builtin_ntp_servers=127.4.158.20:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:46869
--superuser_acl=no-such-user with env {}
W20250812 13:58:39.710394 6810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:39.711002 6810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:39.711493 6810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:39.742554 6810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:58:39.742887 6810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:39.743141 6810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:58:39.743379 6810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:58:39.778474 6810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:46869
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:46869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:39.779830 6810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:39.781489 6810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:39.792328 6816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:39.792356 6817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:40.893282 6819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:40.896463 6810 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.104s user 0.381s sys 0.711s
W20250812 13:58:40.896582 6818 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1102 milliseconds
W20250812 13:58:40.896739 6810 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.105s user 0.381s sys 0.711s
I20250812 13:58:40.897033 6810 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:40.898169 6810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:40.900677 6810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:40.902027 6810 hybrid_clock.cc:648] HybridClock initialized: now 1755007120901976 us; error 63 us; skew 500 ppm
I20250812 13:58:40.902824 6810 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:40.910044 6810 webserver.cc:480] Webserver started at http://127.4.158.62:44241/ using document root <none> and password file <none>
I20250812 13:58:40.911350 6810 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:40.911669 6810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:40.912297 6810 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:40.918743 6810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8723008a9f4a45aca3b2694136291ec7"
format_stamp: "Formatted at 2025-08-12 13:58:40 on dist-test-slave-f6np"
I20250812 13:58:40.920218 6810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8723008a9f4a45aca3b2694136291ec7"
format_stamp: "Formatted at 2025-08-12 13:58:40 on dist-test-slave-f6np"
I20250812 13:58:40.929476 6810 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.005s
I20250812 13:58:40.936630 6826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:40.937775 6810 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250812 13:58:40.938194 6810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8723008a9f4a45aca3b2694136291ec7"
format_stamp: "Formatted at 2025-08-12 13:58:40 on dist-test-slave-f6np"
I20250812 13:58:40.938608 6810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:41.019141 6810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:41.020583 6810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:41.020991 6810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:41.088263 6810 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:46869
I20250812 13:58:41.088361 6877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:46869 every 8 connection(s)
I20250812 13:58:41.090914 6810 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:58:41.093734 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 6810
I20250812 13:58:41.094244 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 13:58:41.096789 6878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:41.119210 6878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7: Bootstrap starting.
I20250812 13:58:41.124943 6878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:41.126538 6878 log.cc:826] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:41.130811 6878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7: No bootstrap required, opened a new log
I20250812 13:58:41.148386 6878 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8723008a9f4a45aca3b2694136291ec7" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46869 } }
I20250812 13:58:41.148998 6878 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:41.149196 6878 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8723008a9f4a45aca3b2694136291ec7, State: Initialized, Role: FOLLOWER
I20250812 13:58:41.149767 6878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8723008a9f4a45aca3b2694136291ec7" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46869 } }
I20250812 13:58:41.150256 6878 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:58:41.150470 6878 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:58:41.150714 6878 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:41.154536 6878 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8723008a9f4a45aca3b2694136291ec7" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46869 } }
I20250812 13:58:41.155161 6878 leader_election.cc:304] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8723008a9f4a45aca3b2694136291ec7; no voters:
I20250812 13:58:41.156730 6878 leader_election.cc:290] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:58:41.157528 6883 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:41.159958 6883 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [term 1 LEADER]: Becoming Leader. State: Replica: 8723008a9f4a45aca3b2694136291ec7, State: Running, Role: LEADER
I20250812 13:58:41.160732 6883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8723008a9f4a45aca3b2694136291ec7" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46869 } }
I20250812 13:58:41.161187 6878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:58:41.168370 6884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8723008a9f4a45aca3b2694136291ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8723008a9f4a45aca3b2694136291ec7" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46869 } } }
I20250812 13:58:41.168318 6885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8723008a9f4a45aca3b2694136291ec7. Latest consensus state: current_term: 1 leader_uuid: "8723008a9f4a45aca3b2694136291ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8723008a9f4a45aca3b2694136291ec7" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 46869 } } }
I20250812 13:58:41.169063 6884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:41.169258 6885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:41.183987 6892 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:58:41.195842 6892 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:58:41.210753 6892 catalog_manager.cc:1349] Generated new cluster ID: 0ed72fc60974452fbd4d4650ff87ccbc
I20250812 13:58:41.211066 6892 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:58:41.229676 6892 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 13:58:41.231624 6892 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:58:41.247911 6892 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8723008a9f4a45aca3b2694136291ec7: Generated new TSK 0
I20250812 13:58:41.248973 6892 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 13:58:41.259574 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:0
--local_ip_for_outbound_sockets=127.4.158.1
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46869
--builtin_ntp_servers=127.4.158.20:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 13:58:41.547976 6902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:41.548481 6902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:41.548978 6902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:41.579847 6902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:41.580703 6902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 13:58:41.621640 6902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:41.623329 6902 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:41.625383 6902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:41.638314 6908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:41.639719 6909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:41.642506 6911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:43.056597 6910 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1411 milliseconds
I20250812 13:58:43.056699 6902 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:43.057838 6902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:43.060348 6902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:43.061757 6902 hybrid_clock.cc:648] HybridClock initialized: now 1755007123061714 us; error 61 us; skew 500 ppm
I20250812 13:58:43.062672 6902 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:43.068676 6902 webserver.cc:480] Webserver started at http://127.4.158.1:45287/ using document root <none> and password file <none>
I20250812 13:58:43.069556 6902 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:43.069757 6902 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:43.070243 6902 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:43.074563 6902 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "682cdf11104349be8bc81e24a33f4926"
format_stamp: "Formatted at 2025-08-12 13:58:43 on dist-test-slave-f6np"
I20250812 13:58:43.075610 6902 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "682cdf11104349be8bc81e24a33f4926"
format_stamp: "Formatted at 2025-08-12 13:58:43 on dist-test-slave-f6np"
I20250812 13:58:43.082527 6902 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250812 13:58:43.088354 6918 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:43.089401 6902 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250812 13:58:43.089691 6902 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "682cdf11104349be8bc81e24a33f4926"
format_stamp: "Formatted at 2025-08-12 13:58:43 on dist-test-slave-f6np"
I20250812 13:58:43.090018 6902 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:43.162461 6902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:43.163990 6902 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:43.164422 6902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:43.166832 6902 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:43.170842 6902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:43.171043 6902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:43.171293 6902 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:43.171442 6902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:43.300527 6902 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:33337
I20250812 13:58:43.300628 7030 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:33337 every 8 connection(s)
I20250812 13:58:43.303061 6902 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 13:58:43.309620 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 6902
I20250812 13:58:43.310034 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 13:58:43.316068 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:0
--local_ip_for_outbound_sockets=127.4.158.2
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46869
--builtin_ntp_servers=127.4.158.20:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:58:43.324170 7031 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46869
I20250812 13:58:43.324674 7031 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:43.325969 7031 heartbeater.cc:507] Master 127.4.158.62:46869 requested a full tablet report, sending...
I20250812 13:58:43.328603 6843 ts_manager.cc:194] Registered new tserver with Master: 682cdf11104349be8bc81e24a33f4926 (127.4.158.1:33337)
I20250812 13:58:43.330545 6843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:46761
W20250812 13:58:43.615224 7035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:43.615664 7035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:43.616106 7035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:43.648270 7035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:43.649104 7035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 13:58:43.683661 7035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:43.684963 7035 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:43.686607 7035 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:43.698370 7041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:44.334545 7031 heartbeater.cc:499] Master 127.4.158.62:46869 was elected leader, sending a full tablet report...
W20250812 13:58:43.698964 7042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:44.901953 7044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:44.904816 7043 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1204 milliseconds
W20250812 13:58:44.905550 7035 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.207s user 0.419s sys 0.774s
W20250812 13:58:44.905819 7035 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.208s user 0.419s sys 0.774s
I20250812 13:58:44.906055 7035 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:44.907066 7035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:44.909405 7035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:44.910758 7035 hybrid_clock.cc:648] HybridClock initialized: now 1755007124910728 us; error 37 us; skew 500 ppm
I20250812 13:58:44.911537 7035 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:44.918774 7035 webserver.cc:480] Webserver started at http://127.4.158.2:43789/ using document root <none> and password file <none>
I20250812 13:58:44.919669 7035 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:44.919881 7035 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:44.920322 7035 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:44.924839 7035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6d5ef11e6d94428e9157a6ddb26ff918"
format_stamp: "Formatted at 2025-08-12 13:58:44 on dist-test-slave-f6np"
I20250812 13:58:44.925951 7035 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6d5ef11e6d94428e9157a6ddb26ff918"
format_stamp: "Formatted at 2025-08-12 13:58:44 on dist-test-slave-f6np"
I20250812 13:58:44.933506 7035 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250812 13:58:44.939224 7051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:44.940420 7035 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250812 13:58:44.940750 7035 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6d5ef11e6d94428e9157a6ddb26ff918"
format_stamp: "Formatted at 2025-08-12 13:58:44 on dist-test-slave-f6np"
I20250812 13:58:44.941108 7035 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:45.008459 7035 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:45.009920 7035 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:45.010339 7035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:45.012724 7035 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:45.016747 7035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:45.016948 7035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:45.017189 7035 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:45.017364 7035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:45.145284 7035 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:35879
I20250812 13:58:45.145368 7163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:35879 every 8 connection(s)
I20250812 13:58:45.147783 7035 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 13:58:45.158413 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 7035
I20250812 13:58:45.158771 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 13:58:45.165319 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:0
--local_ip_for_outbound_sockets=127.4.158.3
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46869
--builtin_ntp_servers=127.4.158.20:44237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:58:45.172308 7164 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46869
I20250812 13:58:45.172727 7164 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:45.173727 7164 heartbeater.cc:507] Master 127.4.158.62:46869 requested a full tablet report, sending...
I20250812 13:58:45.175968 6843 ts_manager.cc:194] Registered new tserver with Master: 6d5ef11e6d94428e9157a6ddb26ff918 (127.4.158.2:35879)
I20250812 13:58:45.177600 6843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:43709
W20250812 13:58:45.471318 7168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:45.471839 7168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:45.472330 7168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:45.504207 7168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:45.505076 7168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 13:58:45.540165 7168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:44237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:46869
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:45.541460 7168 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:45.543063 7168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:45.554760 7174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:46.180960 7164 heartbeater.cc:499] Master 127.4.158.62:46869 was elected leader, sending a full tablet report...
W20250812 13:58:45.555258 7175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:46.778136 7177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:46.782158 7176 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1225 milliseconds
W20250812 13:58:46.782897 7168 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.228s user 0.464s sys 0.756s
W20250812 13:58:46.783139 7168 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.229s user 0.464s sys 0.756s
I20250812 13:58:46.783354 7168 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:46.784368 7168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:46.786880 7168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:46.788318 7168 hybrid_clock.cc:648] HybridClock initialized: now 1755007126788256 us; error 77 us; skew 500 ppm
I20250812 13:58:46.789155 7168 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:46.796708 7168 webserver.cc:480] Webserver started at http://127.4.158.3:33707/ using document root <none> and password file <none>
I20250812 13:58:46.797623 7168 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:46.797868 7168 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:46.798350 7168 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:46.802750 7168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0"
format_stamp: "Formatted at 2025-08-12 13:58:46 on dist-test-slave-f6np"
I20250812 13:58:46.803831 7168 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0"
format_stamp: "Formatted at 2025-08-12 13:58:46 on dist-test-slave-f6np"
I20250812 13:58:46.811471 7168 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250812 13:58:46.817263 7184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:46.818506 7168 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250812 13:58:46.818850 7168 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0"
format_stamp: "Formatted at 2025-08-12 13:58:46 on dist-test-slave-f6np"
I20250812 13:58:46.819180 7168 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:46.882719 7168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:46.884152 7168 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:46.884572 7168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:46.887024 7168 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:46.891072 7168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:46.891279 7168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:46.891521 7168 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:46.891702 7168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:47.017911 7168 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:34503
I20250812 13:58:47.018007 7297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:34503 every 8 connection(s)
I20250812 13:58:47.020417 7168 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 13:58:47.023522 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 7168
I20250812 13:58:47.024154 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 13:58:47.043196 7298 heartbeater.cc:344] Connected to a master server at 127.4.158.62:46869
I20250812 13:58:47.043682 7298 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:47.044965 7298 heartbeater.cc:507] Master 127.4.158.62:46869 requested a full tablet report, sending...
I20250812 13:58:47.046931 6843 ts_manager.cc:194] Registered new tserver with Master: ecd4ba0bbce349cf96322f7d69fa0ae0 (127.4.158.3:34503)
I20250812 13:58:47.048095 6843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:57669
I20250812 13:58:47.058508 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:58:47.091234 6843 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49396:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250812 13:58:47.119993 6843 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 13:58:47.183934 7233 tablet_service.cc:1468] Processing CreateTablet for tablet 5c31d44542864d0298f12eda61c90bde (DEFAULT_TABLE table=TestTable [id=6d6aa162609e4769a1eb77a7e134f797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:47.186028 7233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c31d44542864d0298f12eda61c90bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:47.190443 6966 tablet_service.cc:1468] Processing CreateTablet for tablet 5c31d44542864d0298f12eda61c90bde (DEFAULT_TABLE table=TestTable [id=6d6aa162609e4769a1eb77a7e134f797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:47.190946 7099 tablet_service.cc:1468] Processing CreateTablet for tablet 5c31d44542864d0298f12eda61c90bde (DEFAULT_TABLE table=TestTable [id=6d6aa162609e4769a1eb77a7e134f797]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:47.191892 6966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c31d44542864d0298f12eda61c90bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:47.192735 7099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c31d44542864d0298f12eda61c90bde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:47.212965 7317 tablet_bootstrap.cc:492] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926: Bootstrap starting.
I20250812 13:58:47.215404 7318 tablet_bootstrap.cc:492] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0: Bootstrap starting.
I20250812 13:58:47.219640 7319 tablet_bootstrap.cc:492] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918: Bootstrap starting.
I20250812 13:58:47.222030 7317 tablet_bootstrap.cc:654] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:47.222666 7318 tablet_bootstrap.cc:654] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:47.224336 7318 log.cc:826] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:47.224401 7317 log.cc:826] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:47.226976 7319 tablet_bootstrap.cc:654] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:47.229343 7319 log.cc:826] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:47.229668 7317 tablet_bootstrap.cc:492] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926: No bootstrap required, opened a new log
I20250812 13:58:47.229967 7318 tablet_bootstrap.cc:492] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0: No bootstrap required, opened a new log
I20250812 13:58:47.230116 7317 ts_tablet_manager.cc:1397] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20250812 13:58:47.230443 7318 ts_tablet_manager.cc:1397] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20250812 13:58:47.234841 7319 tablet_bootstrap.cc:492] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918: No bootstrap required, opened a new log
I20250812 13:58:47.235328 7319 ts_tablet_manager.cc:1397] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.001s
I20250812 13:58:47.248687 7317 raft_consensus.cc:357] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.249374 7317 raft_consensus.cc:383] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:47.249639 7317 raft_consensus.cc:738] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682cdf11104349be8bc81e24a33f4926, State: Initialized, Role: FOLLOWER
I20250812 13:58:47.250465 7317 consensus_queue.cc:260] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.255148 7317 ts_tablet_manager.cc:1428] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926: Time spent starting tablet: real 0.025s user 0.022s sys 0.001s
I20250812 13:58:47.255764 7318 raft_consensus.cc:357] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.256655 7318 raft_consensus.cc:383] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:47.256964 7318 raft_consensus.cc:738] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd4ba0bbce349cf96322f7d69fa0ae0, State: Initialized, Role: FOLLOWER
I20250812 13:58:47.257795 7318 consensus_queue.cc:260] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.261647 7298 heartbeater.cc:499] Master 127.4.158.62:46869 was elected leader, sending a full tablet report...
I20250812 13:58:47.261210 7319 raft_consensus.cc:357] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.262189 7319 raft_consensus.cc:383] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:47.262535 7319 raft_consensus.cc:738] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5ef11e6d94428e9157a6ddb26ff918, State: Initialized, Role: FOLLOWER
I20250812 13:58:47.263594 7319 consensus_queue.cc:260] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.263274 7318 ts_tablet_manager.cc:1428] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0: Time spent starting tablet: real 0.033s user 0.032s sys 0.000s
I20250812 13:58:47.267685 7319 ts_tablet_manager.cc:1428] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918: Time spent starting tablet: real 0.032s user 0.026s sys 0.004s
W20250812 13:58:47.274758 7299 tablet.cc:2378] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 13:58:47.311219 7032 tablet.cc:2378] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:58:47.323360 7324 raft_consensus.cc:491] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:58:47.323863 7324 raft_consensus.cc:513] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.326222 7324 leader_election.cc:290] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d5ef11e6d94428e9157a6ddb26ff918 (127.4.158.2:35879), 682cdf11104349be8bc81e24a33f4926 (127.4.158.1:33337)
I20250812 13:58:47.334582 7119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c31d44542864d0298f12eda61c90bde" candidate_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" is_pre_election: true
I20250812 13:58:47.335345 7119 raft_consensus.cc:2466] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecd4ba0bbce349cf96322f7d69fa0ae0 in term 0.
I20250812 13:58:47.336381 6986 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c31d44542864d0298f12eda61c90bde" candidate_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682cdf11104349be8bc81e24a33f4926" is_pre_election: true
I20250812 13:58:47.336477 7187 leader_election.cc:304] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5ef11e6d94428e9157a6ddb26ff918, ecd4ba0bbce349cf96322f7d69fa0ae0; no voters:
I20250812 13:58:47.337042 6986 raft_consensus.cc:2466] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecd4ba0bbce349cf96322f7d69fa0ae0 in term 0.
I20250812 13:58:47.337182 7324 raft_consensus.cc:2802] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 13:58:47.337457 7324 raft_consensus.cc:491] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:58:47.337725 7324 raft_consensus.cc:3058] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:47.341997 7324 raft_consensus.cc:513] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.343369 7324 leader_election.cc:290] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [CANDIDATE]: Term 1 election: Requested vote from peers 6d5ef11e6d94428e9157a6ddb26ff918 (127.4.158.2:35879), 682cdf11104349be8bc81e24a33f4926 (127.4.158.1:33337)
I20250812 13:58:47.344128 7119 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c31d44542864d0298f12eda61c90bde" candidate_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5ef11e6d94428e9157a6ddb26ff918"
I20250812 13:58:47.344350 6986 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c31d44542864d0298f12eda61c90bde" candidate_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "682cdf11104349be8bc81e24a33f4926"
I20250812 13:58:47.344676 7119 raft_consensus.cc:3058] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:47.344902 6986 raft_consensus.cc:3058] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:47.351584 6986 raft_consensus.cc:2466] T 5c31d44542864d0298f12eda61c90bde P 682cdf11104349be8bc81e24a33f4926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecd4ba0bbce349cf96322f7d69fa0ae0 in term 1.
I20250812 13:58:47.351584 7119 raft_consensus.cc:2466] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecd4ba0bbce349cf96322f7d69fa0ae0 in term 1.
I20250812 13:58:47.352785 7186 leader_election.cc:304] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682cdf11104349be8bc81e24a33f4926, ecd4ba0bbce349cf96322f7d69fa0ae0; no voters:
I20250812 13:58:47.353596 7324 raft_consensus.cc:2802] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:47.355242 7324 raft_consensus.cc:695] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [term 1 LEADER]: Becoming Leader. State: Replica: ecd4ba0bbce349cf96322f7d69fa0ae0, State: Running, Role: LEADER
I20250812 13:58:47.356103 7324 consensus_queue.cc:237] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } }
I20250812 13:58:47.365355 6843 catalog_manager.cc:5582] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecd4ba0bbce349cf96322f7d69fa0ae0 (127.4.158.3). New cstate: current_term: 1 leader_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecd4ba0bbce349cf96322f7d69fa0ae0" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 34503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 } health_report { overall_health: UNKNOWN } } }
W20250812 13:58:47.404440 7165 tablet.cc:2378] T 5c31d44542864d0298f12eda61c90bde P 6d5ef11e6d94428e9157a6ddb26ff918: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:58:47.409260 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:58:47.412484 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 682cdf11104349be8bc81e24a33f4926 to finish bootstrapping
I20250812 13:58:47.424598 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6d5ef11e6d94428e9157a6ddb26ff918 to finish bootstrapping
I20250812 13:58:47.435822 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ecd4ba0bbce349cf96322f7d69fa0ae0 to finish bootstrapping
I20250812 13:58:47.823470 7324 consensus_queue.cc:1035] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5ef11e6d94428e9157a6ddb26ff918" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 35879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 13:58:47.846730 7324 consensus_queue.cc:1035] T 5c31d44542864d0298f12eda61c90bde P ecd4ba0bbce349cf96322f7d69fa0ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "682cdf11104349be8bc81e24a33f4926" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 33337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250812 13:58:49.027688 6843 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:49420
I20250812 13:58:50.108340 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 6902
I20250812 13:58:50.129367 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 7035
I20250812 13:58:50.151932 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 7168
I20250812 13:58:50.174460 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 6810
2025-08-12T13:58:50Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10844 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250812 13:58:50.225085 4728 test_util.cc:276] Using random seed: 1888700835
I20250812 13:58:50.229212 4728 ts_itest-base.cc:115] Starting cluster with:
I20250812 13:58:50.229403 4728 ts_itest-base.cc:116] --------------
I20250812 13:58:50.229561 4728 ts_itest-base.cc:117] 3 tablet servers
I20250812 13:58:50.229694 4728 ts_itest-base.cc:118] 3 replicas per TS
I20250812 13:58:50.229820 4728 ts_itest-base.cc:119] --------------
2025-08-12T13:58:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T13:58:50Z Disabled control of system clock
I20250812 13:58:50.263793 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:44775
--webserver_interface=127.4.158.62
--webserver_port=0
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:44775 with env {}
W20250812 13:58:50.554625 7364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:50.555173 7364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:50.555577 7364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:50.586359 7364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:58:50.586639 7364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:50.586843 7364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:58:50.587047 7364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:58:50.621829 7364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:44775
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:50.623112 7364 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:50.624739 7364 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:50.635205 7370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:50.635816 7371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:51.813755 7373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:51.815907 7372 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1175 milliseconds
W20250812 13:58:51.816687 7364 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.182s user 0.400s sys 0.778s
W20250812 13:58:51.816972 7364 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.182s user 0.400s sys 0.778s
I20250812 13:58:51.817189 7364 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:51.818300 7364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:51.820906 7364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:51.822244 7364 hybrid_clock.cc:648] HybridClock initialized: now 1755007131822200 us; error 62 us; skew 500 ppm
I20250812 13:58:51.823035 7364 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:51.830120 7364 webserver.cc:480] Webserver started at http://127.4.158.62:33517/ using document root <none> and password file <none>
I20250812 13:58:51.831092 7364 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:51.831324 7364 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:51.831771 7364 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:51.836212 7364 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2"
format_stamp: "Formatted at 2025-08-12 13:58:51 on dist-test-slave-f6np"
I20250812 13:58:51.837373 7364 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2"
format_stamp: "Formatted at 2025-08-12 13:58:51 on dist-test-slave-f6np"
I20250812 13:58:51.845330 7364 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250812 13:58:51.851402 7380 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:51.852623 7364 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250812 13:58:51.852948 7364 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2"
format_stamp: "Formatted at 2025-08-12 13:58:51 on dist-test-slave-f6np"
I20250812 13:58:51.853297 7364 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:51.917534 7364 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:51.919025 7364 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:51.919462 7364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:51.986937 7364 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:44775
I20250812 13:58:51.987020 7431 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:44775 every 8 connection(s)
I20250812 13:58:51.989681 7364 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:58:51.992777 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 7364
I20250812 13:58:51.993263 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 13:58:51.995924 7432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:52.019553 7432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap starting.
I20250812 13:58:52.024966 7432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:52.026698 7432 log.cc:826] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:52.030912 7432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: No bootstrap required, opened a new log
I20250812 13:58:52.047477 7432 raft_consensus.cc:357] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:58:52.048172 7432 raft_consensus.cc:383] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:52.048411 7432 raft_consensus.cc:738] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaad71b2ec74b2d806cbaa334eb9ce2, State: Initialized, Role: FOLLOWER
I20250812 13:58:52.049084 7432 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:58:52.049589 7432 raft_consensus.cc:397] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:58:52.049881 7432 raft_consensus.cc:491] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:58:52.050190 7432 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:52.054243 7432 raft_consensus.cc:513] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:58:52.054934 7432 leader_election.cc:304] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecaad71b2ec74b2d806cbaa334eb9ce2; no voters:
I20250812 13:58:52.056510 7432 leader_election.cc:290] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:58:52.057248 7437 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:52.059301 7437 raft_consensus.cc:695] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 1 LEADER]: Becoming Leader. State: Replica: ecaad71b2ec74b2d806cbaa334eb9ce2, State: Running, Role: LEADER
I20250812 13:58:52.059980 7437 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:58:52.060408 7432 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:58:52.067893 7438 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } } }
I20250812 13:58:52.068815 7438 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:52.066758 7439 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecaad71b2ec74b2d806cbaa334eb9ce2. Latest consensus state: current_term: 1 leader_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } } }
I20250812 13:58:52.069458 7439 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: This master's current role is: LEADER
I20250812 13:58:52.077131 7445 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:58:52.089587 7445 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:58:52.111034 7445 catalog_manager.cc:1349] Generated new cluster ID: 46a83685258d4947931226adb763e652
I20250812 13:58:52.111367 7445 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:58:52.134006 7445 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 13:58:52.136023 7445 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:58:52.156347 7445 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Generated new TSK 0
I20250812 13:58:52.157344 7445 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 13:58:52.168462 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:0
--local_ip_for_outbound_sockets=127.4.158.1
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:44775
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 13:58:52.482844 7457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:52.483363 7457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:52.483858 7457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:52.516590 7457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:52.517560 7457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 13:58:52.553579 7457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:52.554931 7457 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:52.556582 7457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:52.569075 7463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:52.574757 7466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:52.571470 7464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:53.905424 7465 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1333 milliseconds
I20250812 13:58:53.905514 7457 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:53.906690 7457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:53.909138 7457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:53.910534 7457 hybrid_clock.cc:648] HybridClock initialized: now 1755007133910478 us; error 73 us; skew 500 ppm
I20250812 13:58:53.911316 7457 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:53.918150 7457 webserver.cc:480] Webserver started at http://127.4.158.1:35391/ using document root <none> and password file <none>
I20250812 13:58:53.919068 7457 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:53.919288 7457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:53.919744 7457 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:53.924259 7457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "73afe7141d5f489d9b82fd2ddf56ceae"
format_stamp: "Formatted at 2025-08-12 13:58:53 on dist-test-slave-f6np"
I20250812 13:58:53.925348 7457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "73afe7141d5f489d9b82fd2ddf56ceae"
format_stamp: "Formatted at 2025-08-12 13:58:53 on dist-test-slave-f6np"
I20250812 13:58:53.933028 7457 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250812 13:58:53.939026 7473 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:53.940097 7457 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250812 13:58:53.940418 7457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73afe7141d5f489d9b82fd2ddf56ceae"
format_stamp: "Formatted at 2025-08-12 13:58:53 on dist-test-slave-f6np"
I20250812 13:58:53.940863 7457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:53.994519 7457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:53.995982 7457 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:53.996424 7457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:53.999320 7457 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:54.003952 7457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:54.004189 7457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:54.004449 7457 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:54.004606 7457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:54.156427 7457 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:35897
I20250812 13:58:54.156596 7585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:35897 every 8 connection(s)
I20250812 13:58:54.159021 7457 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 13:58:54.168330 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 7457
I20250812 13:58:54.168776 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 13:58:54.175796 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:0
--local_ip_for_outbound_sockets=127.4.158.2
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:44775
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:58:54.185446 7586 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:58:54.185942 7586 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:54.186988 7586 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:58:54.189522 7397 ts_manager.cc:194] Registered new tserver with Master: 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897)
I20250812 13:58:54.191462 7397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:51769
W20250812 13:58:54.478680 7590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:54.479205 7590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:54.479707 7590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:54.510561 7590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:54.511425 7590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 13:58:54.546057 7590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:54.547331 7590 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:54.548854 7590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:54.560346 7596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:55.194983 7586 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
W20250812 13:58:54.560961 7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:55.666887 7598 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
W20250812 13:58:55.667858 7599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:55.670806 7590 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.111s user 0.346s sys 0.757s
W20250812 13:58:55.671065 7590 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.111s user 0.346s sys 0.757s
I20250812 13:58:55.671276 7590 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:55.672313 7590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:55.674842 7590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:55.676301 7590 hybrid_clock.cc:648] HybridClock initialized: now 1755007135676238 us; error 73 us; skew 500 ppm
I20250812 13:58:55.677394 7590 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:55.689018 7590 webserver.cc:480] Webserver started at http://127.4.158.2:37581/ using document root <none> and password file <none>
I20250812 13:58:55.689953 7590 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:55.690169 7590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:55.690609 7590 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:55.695057 7590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2270135b49614f74a5e652804cbf4221"
format_stamp: "Formatted at 2025-08-12 13:58:55 on dist-test-slave-f6np"
I20250812 13:58:55.696245 7590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2270135b49614f74a5e652804cbf4221"
format_stamp: "Formatted at 2025-08-12 13:58:55 on dist-test-slave-f6np"
I20250812 13:58:55.703770 7590 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250812 13:58:55.709676 7606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:55.710851 7590 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250812 13:58:55.711175 7590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2270135b49614f74a5e652804cbf4221"
format_stamp: "Formatted at 2025-08-12 13:58:55 on dist-test-slave-f6np"
I20250812 13:58:55.711516 7590 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:55.774158 7590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:55.775595 7590 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:55.776016 7590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:55.778430 7590 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:55.782366 7590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:55.782581 7590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:55.782810 7590 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:55.782960 7590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:55.907888 7590 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:41823
I20250812 13:58:55.907990 7718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:41823 every 8 connection(s)
I20250812 13:58:55.910462 7590 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 13:58:55.916221 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 7590
I20250812 13:58:55.916615 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 13:58:55.922088 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:0
--local_ip_for_outbound_sockets=127.4.158.3
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:44775
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:58:55.932214 7719 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:58:55.932664 7719 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:55.933668 7719 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:58:55.935854 7397 ts_manager.cc:194] Registered new tserver with Master: 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823)
I20250812 13:58:55.937245 7397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:53871
W20250812 13:58:56.211706 7723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:58:56.212217 7723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:58:56.212698 7723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:58:56.243484 7723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:58:56.244350 7723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 13:58:56.278528 7723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:58:56.279824 7723 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:58:56.281365 7723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:58:56.292141 7729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:58:56.940258 7719 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
W20250812 13:58:56.296705 7732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:56.293673 7730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:58:57.365731 7731 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 13:58:57.365890 7723 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:58:57.369330 7723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:58:57.371816 7723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:58:57.373216 7723 hybrid_clock.cc:648] HybridClock initialized: now 1755007137373159 us; error 75 us; skew 500 ppm
I20250812 13:58:57.374032 7723 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:58:57.380079 7723 webserver.cc:480] Webserver started at http://127.4.158.3:43395/ using document root <none> and password file <none>
I20250812 13:58:57.380928 7723 fs_manager.cc:362] Metadata directory not provided
I20250812 13:58:57.381126 7723 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:58:57.381551 7723 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:58:57.385737 7723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a98794df53b74eeca7d91b44f01ab94b"
format_stamp: "Formatted at 2025-08-12 13:58:57 on dist-test-slave-f6np"
I20250812 13:58:57.386801 7723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a98794df53b74eeca7d91b44f01ab94b"
format_stamp: "Formatted at 2025-08-12 13:58:57 on dist-test-slave-f6np"
I20250812 13:58:57.393524 7723 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250812 13:58:57.398761 7740 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:57.399684 7723 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250812 13:58:57.399976 7723 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a98794df53b74eeca7d91b44f01ab94b"
format_stamp: "Formatted at 2025-08-12 13:58:57 on dist-test-slave-f6np"
I20250812 13:58:57.400318 7723 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:58:57.446213 7723 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:58:57.447696 7723 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:58:57.448107 7723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:58:57.450482 7723 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:58:57.454568 7723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:58:57.454757 7723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:57.455016 7723 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:58:57.455157 7723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:58:57.579092 7723 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:40737
I20250812 13:58:57.579205 7852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:40737 every 8 connection(s)
I20250812 13:58:57.581575 7723 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 13:58:57.586424 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 7723
I20250812 13:58:57.587087 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 13:58:57.601166 7853 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:58:57.601573 7853 heartbeater.cc:461] Registering TS with master...
I20250812 13:58:57.602527 7853 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:58:57.604550 7397 ts_manager.cc:194] Registered new tserver with Master: a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737)
I20250812 13:58:57.605948 7397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:43095
I20250812 13:58:57.607143 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:58:57.641696 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:58:57.642046 4728 test_util.cc:276] Using random seed: 1896117803
I20250812 13:58:57.680792 7397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43468:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250812 13:58:57.718618 7788 tablet_service.cc:1468] Processing CreateTablet for tablet 133f2f966d4c454c9aff76354bb58ccd (DEFAULT_TABLE table=TestTable [id=8922451354464827920d61845d11db09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:57.720060 7788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133f2f966d4c454c9aff76354bb58ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:57.738127 7873 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Bootstrap starting.
I20250812 13:58:57.743850 7873 tablet_bootstrap.cc:654] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:57.745683 7873 log.cc:826] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:57.749946 7873 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: No bootstrap required, opened a new log
I20250812 13:58:57.750371 7873 ts_tablet_manager.cc:1397] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20250812 13:58:57.766846 7873 raft_consensus.cc:357] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } }
I20250812 13:58:57.767374 7873 raft_consensus.cc:383] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:57.767618 7873 raft_consensus.cc:738] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98794df53b74eeca7d91b44f01ab94b, State: Initialized, Role: FOLLOWER
I20250812 13:58:57.768316 7873 consensus_queue.cc:260] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } }
I20250812 13:58:57.768796 7873 raft_consensus.cc:397] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:58:57.769078 7873 raft_consensus.cc:491] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:58:57.769380 7873 raft_consensus.cc:3058] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:57.773375 7873 raft_consensus.cc:513] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } }
I20250812 13:58:57.774277 7873 leader_election.cc:304] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a98794df53b74eeca7d91b44f01ab94b; no voters:
I20250812 13:58:57.776068 7873 leader_election.cc:290] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:58:57.776438 7875 raft_consensus.cc:2802] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:57.778589 7875 raft_consensus.cc:695] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 1 LEADER]: Becoming Leader. State: Replica: a98794df53b74eeca7d91b44f01ab94b, State: Running, Role: LEADER
I20250812 13:58:57.779717 7853 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
I20250812 13:58:57.779507 7875 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } }
I20250812 13:58:57.781028 7873 ts_tablet_manager.cc:1428] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Time spent starting tablet: real 0.030s user 0.020s sys 0.010s
I20250812 13:58:57.791647 7397 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b reported cstate change: term changed from 0 to 1, leader changed from <none> to a98794df53b74eeca7d91b44f01ab94b (127.4.158.3). New cstate: current_term: 1 leader_uuid: "a98794df53b74eeca7d91b44f01ab94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } health_report { overall_health: HEALTHY } } }
I20250812 13:58:58.014319 4728 test_util.cc:276] Using random seed: 1896490058
I20250812 13:58:58.036813 7395 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43472:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250812 13:58:58.065271 7521 tablet_service.cc:1468] Processing CreateTablet for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 (DEFAULT_TABLE table=TestTable1 [id=c7b9c81f08684840975121681898b14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:58.066690 7521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:58.084038 7894 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap starting.
I20250812 13:58:58.089573 7894 tablet_bootstrap.cc:654] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:58.091228 7894 log.cc:826] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:58.095265 7894 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: No bootstrap required, opened a new log
I20250812 13:58:58.095625 7894 ts_tablet_manager.cc:1397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250812 13:58:58.112402 7894 raft_consensus.cc:357] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } }
I20250812 13:58:58.112923 7894 raft_consensus.cc:383] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:58.113116 7894 raft_consensus.cc:738] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73afe7141d5f489d9b82fd2ddf56ceae, State: Initialized, Role: FOLLOWER
I20250812 13:58:58.113692 7894 consensus_queue.cc:260] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } }
I20250812 13:58:58.114213 7894 raft_consensus.cc:397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:58:58.114444 7894 raft_consensus.cc:491] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:58:58.114707 7894 raft_consensus.cc:3058] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:58.118618 7894 raft_consensus.cc:513] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } }
I20250812 13:58:58.119225 7894 leader_election.cc:304] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73afe7141d5f489d9b82fd2ddf56ceae; no voters:
I20250812 13:58:58.120802 7894 leader_election.cc:290] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:58:58.121171 7896 raft_consensus.cc:2802] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:58.124579 7894 ts_tablet_manager.cc:1428] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent starting tablet: real 0.029s user 0.023s sys 0.006s
I20250812 13:58:58.124533 7896 raft_consensus.cc:695] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 1 LEADER]: Becoming Leader. State: Replica: 73afe7141d5f489d9b82fd2ddf56ceae, State: Running, Role: LEADER
I20250812 13:58:58.125346 7896 consensus_queue.cc:237] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } }
I20250812 13:58:58.135679 7395 catalog_manager.cc:5582] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae reported cstate change: term changed from 0 to 1, leader changed from <none> to 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1). New cstate: current_term: 1 leader_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } health_report { overall_health: HEALTHY } } }
I20250812 13:58:58.303004 4728 test_util.cc:276] Using random seed: 1896778756
I20250812 13:58:58.323058 7396 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43480:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250812 13:58:58.350373 7654 tablet_service.cc:1468] Processing CreateTablet for tablet 691e5640a3eb4d208d8f189d92cd044b (DEFAULT_TABLE table=TestTable2 [id=a8f9783451d54c3ea15ba376e8416820]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:58:58.351743 7654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691e5640a3eb4d208d8f189d92cd044b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:58:58.369529 7915 tablet_bootstrap.cc:492] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Bootstrap starting.
I20250812 13:58:58.374977 7915 tablet_bootstrap.cc:654] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Neither blocks nor log segments found. Creating new log.
I20250812 13:58:58.376817 7915 log.cc:826] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Log is configured to *not* fsync() on all Append() calls
I20250812 13:58:58.380761 7915 tablet_bootstrap.cc:492] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: No bootstrap required, opened a new log
I20250812 13:58:58.381104 7915 ts_tablet_manager.cc:1397] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20250812 13:58:58.397553 7915 raft_consensus.cc:357] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:58:58.398109 7915 raft_consensus.cc:383] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:58:58.398319 7915 raft_consensus.cc:738] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Initialized, Role: FOLLOWER
I20250812 13:58:58.398936 7915 consensus_queue.cc:260] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:58:58.399408 7915 raft_consensus.cc:397] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:58:58.399693 7915 raft_consensus.cc:491] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:58:58.399978 7915 raft_consensus.cc:3058] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:58:58.403894 7915 raft_consensus.cc:513] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:58:58.404520 7915 leader_election.cc:304] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2270135b49614f74a5e652804cbf4221; no voters:
I20250812 13:58:58.406195 7915 leader_election.cc:290] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:58:58.406579 7917 raft_consensus.cc:2802] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:58:58.409902 7917 raft_consensus.cc:695] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 1 LEADER]: Becoming Leader. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Running, Role: LEADER
I20250812 13:58:58.409987 7915 ts_tablet_manager.cc:1428] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Time spent starting tablet: real 0.029s user 0.022s sys 0.008s
I20250812 13:58:58.410748 7917 consensus_queue.cc:237] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
W20250812 13:58:58.417308 7720 tablet.cc:2378] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:58:58.419190 7396 catalog_manager.cc:5582] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2270135b49614f74a5e652804cbf4221 (127.4.158.2). New cstate: current_term: 1 leader_uuid: "2270135b49614f74a5e652804cbf4221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } health_report { overall_health: HEALTHY } } }
I20250812 13:58:58.618134 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 7364
W20250812 13:58:58.806401 7853 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:44775: connect: Connection refused (error 111)
W20250812 13:58:59.152173 7586 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:44775: connect: Connection refused (error 111)
W20250812 13:58:59.440856 7719 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:44775: connect: Connection refused (error 111)
I20250812 13:59:03.300065 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 7457
I20250812 13:59:03.318346 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 7590
I20250812 13:59:03.341415 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 7723
I20250812 13:59:03.363791 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:44775
--webserver_interface=127.4.158.62
--webserver_port=33517
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:44775 with env {}
W20250812 13:59:03.662922 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:03.663543 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:03.664011 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:03.695394 7994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:59:03.695710 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:03.695976 7994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:59:03.696208 7994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:59:03.733018 7994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:44775
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=33517
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:03.734395 7994 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:03.735996 7994 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:03.746892 8000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:03.747465 8001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:04.849257 8003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:04.851593 8002 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1100 milliseconds
I20250812 13:59:04.851740 7994 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:04.852977 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:04.855609 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:04.856972 7994 hybrid_clock.cc:648] HybridClock initialized: now 1755007144856928 us; error 49 us; skew 500 ppm
I20250812 13:59:04.857808 7994 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:04.863888 7994 webserver.cc:480] Webserver started at http://127.4.158.62:33517/ using document root <none> and password file <none>
I20250812 13:59:04.864840 7994 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:04.865068 7994 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:04.873015 7994 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250812 13:59:04.877489 8010 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:04.878582 7994 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250812 13:59:04.878930 7994 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2"
format_stamp: "Formatted at 2025-08-12 13:58:51 on dist-test-slave-f6np"
I20250812 13:59:04.880904 7994 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:04.939935 7994 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:04.941454 7994 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:04.941938 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:05.010426 7994 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:44775
I20250812 13:59:05.010488 8061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:44775 every 8 connection(s)
I20250812 13:59:05.013316 7994 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:59:05.020772 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 7994
I20250812 13:59:05.022177 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:35897
--local_ip_for_outbound_sockets=127.4.158.1
--tserver_master_addrs=127.4.158.62:44775
--webserver_port=35391
--webserver_interface=127.4.158.1
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:59:05.023521 8062 sys_catalog.cc:263] Verifying existing consensus state
I20250812 13:59:05.028437 8062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap starting.
I20250812 13:59:05.038650 8062 log.cc:826] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:05.085390 8062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:05.086241 8062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap complete.
I20250812 13:59:05.106792 8062 raft_consensus.cc:357] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:05.108996 8062 raft_consensus.cc:738] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaad71b2ec74b2d806cbaa334eb9ce2, State: Initialized, Role: FOLLOWER
I20250812 13:59:05.109889 8062 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:05.110436 8062 raft_consensus.cc:397] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:59:05.110698 8062 raft_consensus.cc:491] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:59:05.111014 8062 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 2 FOLLOWER]: Advancing to term 3
I20250812 13:59:05.117033 8062 raft_consensus.cc:513] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:05.117702 8062 leader_election.cc:304] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecaad71b2ec74b2d806cbaa334eb9ce2; no voters:
I20250812 13:59:05.119727 8062 leader_election.cc:290] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250812 13:59:05.120136 8066 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 3 FOLLOWER]: Leader election won for term 3
I20250812 13:59:05.123116 8066 raft_consensus.cc:695] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 3 LEADER]: Becoming Leader. State: Replica: ecaad71b2ec74b2d806cbaa334eb9ce2, State: Running, Role: LEADER
I20250812 13:59:05.124038 8066 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:05.124570 8062 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:59:05.134670 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecaad71b2ec74b2d806cbaa334eb9ce2. Latest consensus state: current_term: 3 leader_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } } }
I20250812 13:59:05.134683 8067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } } }
I20250812 13:59:05.135604 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: This master's current role is: LEADER
I20250812 13:59:05.135630 8067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: This master's current role is: LEADER
I20250812 13:59:05.139299 8073 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:59:05.152352 8073 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=3d7f99971cb54e9aba66929ceb029c51]
I20250812 13:59:05.154147 8073 catalog_manager.cc:671] Loaded metadata for table TestTable [id=676ea4b27c53477cae5da5e680e3aa60]
I20250812 13:59:05.155884 8073 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=a8f9783451d54c3ea15ba376e8416820]
I20250812 13:59:05.163940 8073 tablet_loader.cc:96] loaded metadata for tablet 133f2f966d4c454c9aff76354bb58ccd (table TestTable [id=676ea4b27c53477cae5da5e680e3aa60])
I20250812 13:59:05.165344 8073 tablet_loader.cc:96] loaded metadata for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 (table TestTable1 [id=3d7f99971cb54e9aba66929ceb029c51])
I20250812 13:59:05.166771 8073 tablet_loader.cc:96] loaded metadata for tablet 691e5640a3eb4d208d8f189d92cd044b (table TestTable2 [id=a8f9783451d54c3ea15ba376e8416820])
I20250812 13:59:05.168332 8073 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:59:05.173648 8073 catalog_manager.cc:1261] Loaded cluster ID: 46a83685258d4947931226adb763e652
I20250812 13:59:05.174026 8073 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:59:05.182513 8073 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:59:05.188045 8073 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Loaded TSK: 0
I20250812 13:59:05.189895 8073 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250812 13:59:05.353224 8064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:05.353755 8064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:05.354316 8064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:05.386930 8064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:05.387827 8064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 13:59:05.423694 8064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:35897
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=35391
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:05.425133 8064 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:05.427474 8064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:05.444295 8089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:05.445310 8090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:06.821691 8064 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.379s user 0.527s sys 0.848s
W20250812 13:59:06.822130 8064 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.379s user 0.531s sys 0.848s
I20250812 13:59:06.824572 8064 server_base.cc:1047] running on GCE node
W20250812 13:59:06.826100 8094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:06.827323 8064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:06.829919 8064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:06.831382 8064 hybrid_clock.cc:648] HybridClock initialized: now 1755007146831322 us; error 74 us; skew 500 ppm
I20250812 13:59:06.832243 8064 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:06.840178 8064 webserver.cc:480] Webserver started at http://127.4.158.1:35391/ using document root <none> and password file <none>
I20250812 13:59:06.841138 8064 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:06.841380 8064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:06.849813 8064 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250812 13:59:06.854816 8099 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:06.855953 8064 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250812 13:59:06.856302 8064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73afe7141d5f489d9b82fd2ddf56ceae"
format_stamp: "Formatted at 2025-08-12 13:58:53 on dist-test-slave-f6np"
I20250812 13:59:06.858557 8064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:06.926939 8064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:06.928467 8064 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:06.928967 8064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:06.932152 8064 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:06.938900 8106 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 13:59:06.947002 8064 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 13:59:06.947275 8064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250812 13:59:06.947566 8064 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 13:59:06.952519 8064 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 13:59:06.952874 8064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.002s
I20250812 13:59:06.953969 8106 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap starting.
I20250812 13:59:07.027684 8106 log.cc:826] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:07.151384 8106 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:07.152410 8106 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap complete.
I20250812 13:59:07.154250 8106 ts_tablet_manager.cc:1397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent bootstrapping tablet: real 0.201s user 0.156s sys 0.037s
I20250812 13:59:07.158303 8064 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:35897
I20250812 13:59:07.158509 8214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:35897 every 8 connection(s)
I20250812 13:59:07.161892 8064 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 13:59:07.169893 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 8064
I20250812 13:59:07.171995 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:41823
--local_ip_for_outbound_sockets=127.4.158.2
--tserver_master_addrs=127.4.158.62:44775
--webserver_port=37581
--webserver_interface=127.4.158.2
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:59:07.177328 8106 raft_consensus.cc:357] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } }
I20250812 13:59:07.180943 8106 raft_consensus.cc:738] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73afe7141d5f489d9b82fd2ddf56ceae, State: Initialized, Role: FOLLOWER
I20250812 13:59:07.182093 8106 consensus_queue.cc:260] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } }
I20250812 13:59:07.182873 8106 raft_consensus.cc:397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:59:07.183251 8106 raft_consensus.cc:491] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:59:07.183672 8106 raft_consensus.cc:3058] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 1 FOLLOWER]: Advancing to term 2
I20250812 13:59:07.192660 8106 raft_consensus.cc:513] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } }
I20250812 13:59:07.193758 8106 leader_election.cc:304] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73afe7141d5f489d9b82fd2ddf56ceae; no voters:
I20250812 13:59:07.201622 8106 leader_election.cc:290] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 13:59:07.202211 8219 raft_consensus.cc:2802] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Leader election won for term 2
I20250812 13:59:07.209314 8219 raft_consensus.cc:695] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEADER]: Becoming Leader. State: Replica: 73afe7141d5f489d9b82fd2ddf56ceae, State: Running, Role: LEADER
I20250812 13:59:07.210861 8219 consensus_queue.cc:237] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } }
I20250812 13:59:07.218520 8106 ts_tablet_manager.cc:1428] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent starting tablet: real 0.064s user 0.053s sys 0.008s
I20250812 13:59:07.236531 8215 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:59:07.237094 8215 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:07.238433 8215 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:59:07.243467 8027 ts_manager.cc:194] Registered new tserver with Master: 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897)
I20250812 13:59:07.247749 8027 catalog_manager.cc:5582] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae reported cstate change: term changed from 0 to 2, leader changed from <none> to 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1), VOTER 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) added. New cstate: current_term: 2 leader_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } health_report { overall_health: HEALTHY } } }
W20250812 13:59:07.294783 8027 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 with cas_config_opid_index -1: no extra replica candidate found for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 (table TestTable1 [id=3d7f99971cb54e9aba66929ceb029c51]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250812 13:59:07.297142 8027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:52117
I20250812 13:59:07.301589 8215 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
W20250812 13:59:07.525107 8222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:07.525585 8222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:07.526140 8222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:07.557948 8222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:07.558789 8222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 13:59:07.593703 8222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:41823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=37581
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:07.595070 8222 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:07.596693 8222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:07.609756 8235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:07.610507 8234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:08.717825 8222 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.110s user 0.492s sys 0.613s
W20250812 13:59:08.718326 8222 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.111s user 0.496s sys 0.613s
I20250812 13:59:08.722621 8222 server_base.cc:1047] running on GCE node
W20250812 13:59:08.723096 8239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:08.724663 8222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:08.733090 8222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:08.734794 8222 hybrid_clock.cc:648] HybridClock initialized: now 1755007148734747 us; error 65 us; skew 500 ppm
I20250812 13:59:08.735985 8222 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:08.744722 8222 webserver.cc:480] Webserver started at http://127.4.158.2:37581/ using document root <none> and password file <none>
I20250812 13:59:08.746055 8222 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:08.746372 8222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:08.757530 8222 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20250812 13:59:08.763449 8244 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:08.764719 8222 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250812 13:59:08.765134 8222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2270135b49614f74a5e652804cbf4221"
format_stamp: "Formatted at 2025-08-12 13:58:55 on dist-test-slave-f6np"
I20250812 13:59:08.768092 8222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:08.840065 8222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:08.841611 8222 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:08.842048 8222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:08.844607 8222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:08.850165 8251 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 13:59:08.857592 8222 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 13:59:08.857825 8222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250812 13:59:08.858095 8222 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 13:59:08.862713 8222 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 13:59:08.862897 8222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250812 13:59:08.863265 8251 tablet_bootstrap.cc:492] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Bootstrap starting.
I20250812 13:59:08.916759 8251 log.cc:826] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:08.999671 8251 tablet_bootstrap.cc:492] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:09.000474 8251 tablet_bootstrap.cc:492] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Bootstrap complete.
I20250812 13:59:09.001629 8251 ts_tablet_manager.cc:1397] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Time spent bootstrapping tablet: real 0.139s user 0.105s sys 0.028s
I20250812 13:59:09.020193 8251 raft_consensus.cc:357] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:59:09.023409 8251 raft_consensus.cc:738] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Initialized, Role: FOLLOWER
I20250812 13:59:09.024449 8251 consensus_queue.cc:260] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:59:09.025115 8251 raft_consensus.cc:397] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:59:09.025480 8251 raft_consensus.cc:491] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:59:09.026000 8251 raft_consensus.cc:3058] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 1 FOLLOWER]: Advancing to term 2
I20250812 13:59:09.033804 8251 raft_consensus.cc:513] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:59:09.034729 8251 leader_election.cc:304] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2270135b49614f74a5e652804cbf4221; no voters:
I20250812 13:59:09.037585 8222 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:41823
I20250812 13:59:09.037756 8359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:41823 every 8 connection(s)
I20250812 13:59:09.038623 8251 leader_election.cc:290] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 13:59:09.039139 8358 raft_consensus.cc:2802] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 13:59:09.041260 8222 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 13:59:09.042408 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 8222
I20250812 13:59:09.042559 8251 ts_tablet_manager.cc:1428] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Time spent starting tablet: real 0.041s user 0.037s sys 0.005s
I20250812 13:59:09.043928 8358 raft_consensus.cc:695] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 2 LEADER]: Becoming Leader. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Running, Role: LEADER
I20250812 13:59:09.044253 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:40737
--local_ip_for_outbound_sockets=127.4.158.3
--tserver_master_addrs=127.4.158.62:44775
--webserver_port=43395
--webserver_interface=127.4.158.3
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:59:09.045199 8358 consensus_queue.cc:237] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:59:09.067938 8362 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:59:09.068482 8362 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:09.069839 8362 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:59:09.073980 8027 ts_manager.cc:194] Registered new tserver with Master: 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823)
I20250812 13:59:09.074954 8027 catalog_manager.cc:5582] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "2270135b49614f74a5e652804cbf4221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } health_report { overall_health: HEALTHY } } }
I20250812 13:59:09.085393 8027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:49991
I20250812 13:59:09.089483 8362 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
W20250812 13:59:09.353962 8366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:09.354447 8366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:09.354889 8366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:09.386984 8366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:09.387799 8366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 13:59:09.423309 8366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:40737
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=43395
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:09.424558 8366 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:09.426190 8366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:09.438843 8378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:09.634222 8170 consensus_queue.cc:237] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:09.658953 8388 raft_consensus.cc:2953] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 2270135b49614f74a5e652804cbf4221 (127.4.158.2) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } } }
I20250812 13:59:09.671805 8011 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
W20250812 13:59:09.678370 8102 consensus_peers.cc:489] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae -> Peer 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823): Couldn't send request to peer 2270135b49614f74a5e652804cbf4221. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bf9ef4b2e5241ca8c8b86a9fe3dc829. This is attempt 1: this message will repeat every 5th retry.
I20250812 13:59:09.680471 8027 catalog_manager.cc:5582] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae reported cstate change: config changed from index -1 to 9, NON_VOTER 2270135b49614f74a5e652804cbf4221 (127.4.158.2) added. New cstate: current_term: 2 leader_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250812 13:59:10.100560 8394 ts_tablet_manager.cc:927] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Initiating tablet copy from peer 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897)
I20250812 13:59:10.111025 8394 tablet_copy_client.cc:323] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: tablet copy: Beginning tablet copy session from remote peer at address 127.4.158.1:35897
I20250812 13:59:10.137161 8190 tablet_copy_service.cc:140] P 73afe7141d5f489d9b82fd2ddf56ceae: Received BeginTabletCopySession request for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 from peer 2270135b49614f74a5e652804cbf4221 ({username='slave'} at 127.4.158.2:41761)
I20250812 13:59:10.137992 8190 tablet_copy_service.cc:161] P 73afe7141d5f489d9b82fd2ddf56ceae: Beginning new tablet copy session on tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 from peer 2270135b49614f74a5e652804cbf4221 at {username='slave'} at 127.4.158.2:41761: session id = 2270135b49614f74a5e652804cbf4221-4bf9ef4b2e5241ca8c8b86a9fe3dc829
I20250812 13:59:10.146714 8190 tablet_copy_source_session.cc:215] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 13:59:10.152232 8394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:59:10.170394 8394 tablet_copy_client.cc:806] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: tablet copy: Starting download of 0 data blocks...
I20250812 13:59:10.171244 8394 tablet_copy_client.cc:670] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: tablet copy: Starting download of 1 WAL segments...
I20250812 13:59:10.180403 8394 tablet_copy_client.cc:538] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 13:59:10.191741 8394 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Bootstrap starting.
I20250812 13:59:10.335042 8394 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:10.336014 8394 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Bootstrap complete.
I20250812 13:59:10.336678 8394 ts_tablet_manager.cc:1397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Time spent bootstrapping tablet: real 0.145s user 0.132s sys 0.006s
I20250812 13:59:10.339452 8394 raft_consensus.cc:357] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:10.340158 8394 raft_consensus.cc:738] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Initialized, Role: LEARNER
I20250812 13:59:10.340762 8394 consensus_queue.cc:260] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:10.344246 8394 ts_tablet_manager.cc:1428] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20250812 13:59:10.346774 8190 tablet_copy_service.cc:342] P 73afe7141d5f489d9b82fd2ddf56ceae: Request end of tablet copy session 2270135b49614f74a5e652804cbf4221-4bf9ef4b2e5241ca8c8b86a9fe3dc829 received from {username='slave'} at 127.4.158.2:41761
I20250812 13:59:10.347337 8190 tablet_copy_service.cc:434] P 73afe7141d5f489d9b82fd2ddf56ceae: ending tablet copy session 2270135b49614f74a5e652804cbf4221-4bf9ef4b2e5241ca8c8b86a9fe3dc829 on tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 with peer 2270135b49614f74a5e652804cbf4221
W20250812 13:59:09.439327 8379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:10.601634 8381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:10.604480 8380 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1165 milliseconds
I20250812 13:59:10.604571 8366 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:10.605717 8366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:10.607954 8366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:10.609298 8366 hybrid_clock.cc:648] HybridClock initialized: now 1755007150609261 us; error 54 us; skew 500 ppm
I20250812 13:59:10.610117 8366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:10.616190 8366 webserver.cc:480] Webserver started at http://127.4.158.3:43395/ using document root <none> and password file <none>
I20250812 13:59:10.617128 8366 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:10.617377 8366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:10.625321 8366 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250812 13:59:10.630529 8404 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:10.632586 8366 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250812 13:59:10.632856 8314 raft_consensus.cc:1215] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250812 13:59:10.632959 8366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a98794df53b74eeca7d91b44f01ab94b"
format_stamp: "Formatted at 2025-08-12 13:58:57 on dist-test-slave-f6np"
I20250812 13:59:10.635811 8366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:10.684816 8366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:10.686476 8366 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:10.686942 8366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:10.689453 8366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:10.695639 8413 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 13:59:10.703372 8366 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 13:59:10.703620 8366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250812 13:59:10.703892 8366 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 13:59:10.708832 8366 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 13:59:10.709105 8366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250812 13:59:10.709503 8413 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Bootstrap starting.
I20250812 13:59:10.784865 8413 log.cc:826] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:10.878037 8366 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:40737
I20250812 13:59:10.878208 8520 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:40737 every 8 connection(s)
I20250812 13:59:10.880555 8366 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 13:59:10.881481 8413 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:10.882257 8413 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Bootstrap complete.
I20250812 13:59:10.883409 8413 ts_tablet_manager.cc:1397] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Time spent bootstrapping tablet: real 0.174s user 0.137s sys 0.035s
I20250812 13:59:10.887836 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 8366
I20250812 13:59:10.904405 8413 raft_consensus.cc:357] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } }
I20250812 13:59:10.906869 8413 raft_consensus.cc:738] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98794df53b74eeca7d91b44f01ab94b, State: Initialized, Role: FOLLOWER
I20250812 13:59:10.908114 8413 consensus_queue.cc:260] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } }
I20250812 13:59:10.908728 8413 raft_consensus.cc:397] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:59:10.909124 8413 raft_consensus.cc:491] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:59:10.909572 8413 raft_consensus.cc:3058] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 1 FOLLOWER]: Advancing to term 2
I20250812 13:59:10.912036 8521 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:59:10.912523 8521 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:10.913714 8521 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:59:10.916108 8413 raft_consensus.cc:513] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } }
I20250812 13:59:10.916756 8413 leader_election.cc:304] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a98794df53b74eeca7d91b44f01ab94b; no voters:
I20250812 13:59:10.918819 8413 leader_election.cc:290] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 13:59:10.919333 8527 raft_consensus.cc:2802] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Leader election won for term 2
I20250812 13:59:10.922725 8527 raft_consensus.cc:695] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 LEADER]: Becoming Leader. State: Replica: a98794df53b74eeca7d91b44f01ab94b, State: Running, Role: LEADER
I20250812 13:59:10.923205 8413 ts_tablet_manager.cc:1428] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Time spent starting tablet: real 0.039s user 0.026s sys 0.008s
I20250812 13:59:10.923581 8527 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } }
I20250812 13:59:10.925805 8026 ts_manager.cc:194] Registered new tserver with Master: a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737)
I20250812 13:59:10.928799 8026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:48301
I20250812 13:59:10.929338 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:59:10.932673 8521 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
I20250812 13:59:10.934096 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:59:10.935848 8026 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b reported cstate change: term changed from 0 to 2, leader changed from <none> to a98794df53b74eeca7d91b44f01ab94b (127.4.158.3), VOTER a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) added. New cstate: current_term: 2 leader_uuid: "a98794df53b74eeca7d91b44f01ab94b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } health_report { overall_health: HEALTHY } } }
W20250812 13:59:10.936786 4728 ts_itest-base.cc:209] found only 0 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" }
I20250812 13:59:10.960574 8476 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } }
I20250812 13:59:10.964084 8529 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } } }
I20250812 13:59:10.972267 8014 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 133f2f966d4c454c9aff76354bb58ccd with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250812 13:59:10.974260 8407 consensus_peers.cc:489] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b -> Peer 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897): Couldn't send request to peer 73afe7141d5f489d9b82fd2ddf56ceae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 133f2f966d4c454c9aff76354bb58ccd. This is attempt 1: this message will repeat every 5th retry.
I20250812 13:59:10.975055 8026 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b reported cstate change: config changed from index -1 to 10, NON_VOTER 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) added. New cstate: current_term: 2 leader_uuid: "a98794df53b74eeca7d91b44f01ab94b" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250812 13:59:10.984638 8476 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:10.987926 8529 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2270135b49614f74a5e652804cbf4221 (127.4.158.2) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } } }
W20250812 13:59:10.989584 8407 consensus_peers.cc:489] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b -> Peer 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897): Couldn't send request to peer 73afe7141d5f489d9b82fd2ddf56ceae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 133f2f966d4c454c9aff76354bb58ccd. This is attempt 1: this message will repeat every 5th retry.
I20250812 13:59:10.995784 8014 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 133f2f966d4c454c9aff76354bb58ccd with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250812 13:59:10.998780 8409 consensus_peers.cc:489] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b -> Peer 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823): Couldn't send request to peer 2270135b49614f74a5e652804cbf4221. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 133f2f966d4c454c9aff76354bb58ccd. This is attempt 1: this message will repeat every 5th retry.
I20250812 13:59:10.998677 8026 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b reported cstate change: config changed from index 10 to 11, NON_VOTER 2270135b49614f74a5e652804cbf4221 (127.4.158.2) added. New cstate: current_term: 2 leader_uuid: "a98794df53b74eeca7d91b44f01ab94b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250812 13:59:11.102300 8406 raft_consensus.cc:1062] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: attempting to promote NON_VOTER 2270135b49614f74a5e652804cbf4221 to VOTER
I20250812 13:59:11.103693 8406 consensus_queue.cc:237] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:11.107738 8314 raft_consensus.cc:1273] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Refusing update from remote peer 73afe7141d5f489d9b82fd2ddf56ceae: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250812 13:59:11.108981 8405 consensus_queue.cc:1035] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250812 13:59:11.114984 8406 raft_consensus.cc:2953] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 2270135b49614f74a5e652804cbf4221 (127.4.158.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:11.116498 8314 raft_consensus.cc:2953] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 2270135b49614f74a5e652804cbf4221 (127.4.158.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:11.126449 8026 catalog_manager.cc:5582] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae reported cstate change: config changed from index 9 to 10, 2270135b49614f74a5e652804cbf4221 (127.4.158.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250812 13:59:11.136035 8170 consensus_queue.cc:237] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: NON_VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: true } }
I20250812 13:59:11.140688 8314 raft_consensus.cc:1273] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Refusing update from remote peer 73afe7141d5f489d9b82fd2ddf56ceae: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250812 13:59:11.141975 8406 consensus_queue.cc:1035] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250812 13:59:11.147712 8405 raft_consensus.cc:2953] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: NON_VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: true } } }
I20250812 13:59:11.149195 8314 raft_consensus.cc:2953] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: NON_VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: true } } }
W20250812 13:59:11.150950 8103 consensus_peers.cc:489] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae -> Peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): Couldn't send request to peer a98794df53b74eeca7d91b44f01ab94b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bf9ef4b2e5241ca8c8b86a9fe3dc829. This is attempt 1: this message will repeat every 5th retry.
I20250812 13:59:11.155771 8011 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250812 13:59:11.158628 8026 catalog_manager.cc:5582] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 reported cstate change: config changed from index 10 to 11, NON_VOTER a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) added. New cstate: current_term: 2 leader_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: NON_VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: true } } }
I20250812 13:59:11.553104 8543 ts_tablet_manager.cc:927] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Initiating tablet copy from peer 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897)
I20250812 13:59:11.554970 8543 tablet_copy_client.cc:323] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.158.1:35897
I20250812 13:59:11.556464 8190 tablet_copy_service.cc:140] P 73afe7141d5f489d9b82fd2ddf56ceae: Received BeginTabletCopySession request for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 from peer a98794df53b74eeca7d91b44f01ab94b ({username='slave'} at 127.4.158.3:58251)
I20250812 13:59:11.556878 8190 tablet_copy_service.cc:161] P 73afe7141d5f489d9b82fd2ddf56ceae: Beginning new tablet copy session on tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 from peer a98794df53b74eeca7d91b44f01ab94b at {username='slave'} at 127.4.158.3:58251: session id = a98794df53b74eeca7d91b44f01ab94b-4bf9ef4b2e5241ca8c8b86a9fe3dc829
I20250812 13:59:11.561209 8190 tablet_copy_source_session.cc:215] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 13:59:11.564071 8543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:59:11.568883 8546 ts_tablet_manager.cc:927] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Initiating tablet copy from peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737)
I20250812 13:59:11.571463 8546 tablet_copy_client.cc:323] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: tablet copy: Beginning tablet copy session from remote peer at address 127.4.158.3:40737
I20250812 13:59:11.573271 8496 tablet_copy_service.cc:140] P a98794df53b74eeca7d91b44f01ab94b: Received BeginTabletCopySession request for tablet 133f2f966d4c454c9aff76354bb58ccd from peer 73afe7141d5f489d9b82fd2ddf56ceae ({username='slave'} at 127.4.158.1:36419)
I20250812 13:59:11.573778 8496 tablet_copy_service.cc:161] P a98794df53b74eeca7d91b44f01ab94b: Beginning new tablet copy session on tablet 133f2f966d4c454c9aff76354bb58ccd from peer 73afe7141d5f489d9b82fd2ddf56ceae at {username='slave'} at 127.4.158.1:36419: session id = 73afe7141d5f489d9b82fd2ddf56ceae-133f2f966d4c454c9aff76354bb58ccd
I20250812 13:59:11.578645 8496 tablet_copy_source_session.cc:215] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 13:59:11.578920 8543 tablet_copy_client.cc:806] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: tablet copy: Starting download of 0 data blocks...
I20250812 13:59:11.579479 8543 tablet_copy_client.cc:670] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: tablet copy: Starting download of 1 WAL segments...
I20250812 13:59:11.582438 8546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133f2f966d4c454c9aff76354bb58ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:59:11.583509 8543 tablet_copy_client.cc:538] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 13:59:11.589505 8543 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Bootstrap starting.
I20250812 13:59:11.592911 8546 tablet_copy_client.cc:806] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: tablet copy: Starting download of 0 data blocks...
I20250812 13:59:11.593472 8546 tablet_copy_client.cc:670] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: tablet copy: Starting download of 1 WAL segments...
I20250812 13:59:11.597652 8546 tablet_copy_client.cc:538] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 13:59:11.603643 8546 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap starting.
I20250812 13:59:11.604032 8548 ts_tablet_manager.cc:927] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Initiating tablet copy from peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737)
I20250812 13:59:11.605683 8548 tablet_copy_client.cc:323] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: tablet copy: Beginning tablet copy session from remote peer at address 127.4.158.3:40737
I20250812 13:59:11.621783 8496 tablet_copy_service.cc:140] P a98794df53b74eeca7d91b44f01ab94b: Received BeginTabletCopySession request for tablet 133f2f966d4c454c9aff76354bb58ccd from peer 2270135b49614f74a5e652804cbf4221 ({username='slave'} at 127.4.158.2:56555)
I20250812 13:59:11.622365 8496 tablet_copy_service.cc:161] P a98794df53b74eeca7d91b44f01ab94b: Beginning new tablet copy session on tablet 133f2f966d4c454c9aff76354bb58ccd from peer 2270135b49614f74a5e652804cbf4221 at {username='slave'} at 127.4.158.2:56555: session id = 2270135b49614f74a5e652804cbf4221-133f2f966d4c454c9aff76354bb58ccd
I20250812 13:59:11.629320 8496 tablet_copy_source_session.cc:215] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Tablet Copy: opened 0 blocks and 1 log segments
I20250812 13:59:11.632350 8548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 133f2f966d4c454c9aff76354bb58ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:59:11.647696 8548 tablet_copy_client.cc:806] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: tablet copy: Starting download of 0 data blocks...
I20250812 13:59:11.648327 8548 tablet_copy_client.cc:670] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: tablet copy: Starting download of 1 WAL segments...
I20250812 13:59:11.652449 8548 tablet_copy_client.cc:538] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250812 13:59:11.661273 8548 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Bootstrap starting.
I20250812 13:59:11.705993 8546 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:11.706640 8546 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap complete.
I20250812 13:59:11.707096 8546 ts_tablet_manager.cc:1397] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent bootstrapping tablet: real 0.104s user 0.090s sys 0.013s
I20250812 13:59:11.708747 8546 raft_consensus.cc:357] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:11.709247 8546 raft_consensus.cc:738] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 73afe7141d5f489d9b82fd2ddf56ceae, State: Initialized, Role: LEARNER
I20250812 13:59:11.709369 8543 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:11.709729 8546 consensus_queue.cc:260] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:11.710179 8543 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Bootstrap complete.
I20250812 13:59:11.710736 8543 ts_tablet_manager.cc:1397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Time spent bootstrapping tablet: real 0.121s user 0.107s sys 0.016s
I20250812 13:59:11.711452 8546 ts_tablet_manager.cc:1428] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20250812 13:59:11.713056 8496 tablet_copy_service.cc:342] P a98794df53b74eeca7d91b44f01ab94b: Request end of tablet copy session 73afe7141d5f489d9b82fd2ddf56ceae-133f2f966d4c454c9aff76354bb58ccd received from {username='slave'} at 127.4.158.1:36419
I20250812 13:59:11.713572 8496 tablet_copy_service.cc:434] P a98794df53b74eeca7d91b44f01ab94b: ending tablet copy session 73afe7141d5f489d9b82fd2ddf56ceae-133f2f966d4c454c9aff76354bb58ccd on tablet 133f2f966d4c454c9aff76354bb58ccd with peer 73afe7141d5f489d9b82fd2ddf56ceae
I20250812 13:59:11.713114 8543 raft_consensus.cc:357] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: NON_VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: true } }
I20250812 13:59:11.714236 8543 raft_consensus.cc:738] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a98794df53b74eeca7d91b44f01ab94b, State: Initialized, Role: LEARNER
I20250812 13:59:11.714834 8543 consensus_queue.cc:260] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: NON_VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: true } }
I20250812 13:59:11.717085 8543 ts_tablet_manager.cc:1428] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250812 13:59:11.718603 8190 tablet_copy_service.cc:342] P 73afe7141d5f489d9b82fd2ddf56ceae: Request end of tablet copy session a98794df53b74eeca7d91b44f01ab94b-4bf9ef4b2e5241ca8c8b86a9fe3dc829 received from {username='slave'} at 127.4.158.3:58251
I20250812 13:59:11.718922 8190 tablet_copy_service.cc:434] P 73afe7141d5f489d9b82fd2ddf56ceae: ending tablet copy session a98794df53b74eeca7d91b44f01ab94b-4bf9ef4b2e5241ca8c8b86a9fe3dc829 on tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 with peer a98794df53b74eeca7d91b44f01ab94b
I20250812 13:59:11.775346 8548 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:11.775971 8548 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Bootstrap complete.
I20250812 13:59:11.776418 8548 ts_tablet_manager.cc:1397] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Time spent bootstrapping tablet: real 0.115s user 0.107s sys 0.004s
I20250812 13:59:11.778010 8548 raft_consensus.cc:357] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:11.778492 8548 raft_consensus.cc:738] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Initialized, Role: LEARNER
I20250812 13:59:11.778899 8548 consensus_queue.cc:260] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: NON_VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: true } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:11.781198 8548 ts_tablet_manager.cc:1428] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250812 13:59:11.782653 8496 tablet_copy_service.cc:342] P a98794df53b74eeca7d91b44f01ab94b: Request end of tablet copy session 2270135b49614f74a5e652804cbf4221-133f2f966d4c454c9aff76354bb58ccd received from {username='slave'} at 127.4.158.2:56555
I20250812 13:59:11.782966 8496 tablet_copy_service.cc:434] P a98794df53b74eeca7d91b44f01ab94b: ending tablet copy session 2270135b49614f74a5e652804cbf4221-133f2f966d4c454c9aff76354bb58ccd on tablet 133f2f966d4c454c9aff76354bb58ccd with peer 2270135b49614f74a5e652804cbf4221
I20250812 13:59:11.916482 8013 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250812 13:59:11.941370 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 73afe7141d5f489d9b82fd2ddf56ceae to finish bootstrapping
I20250812 13:59:11.954380 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2270135b49614f74a5e652804cbf4221 to finish bootstrapping
I20250812 13:59:11.967298 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a98794df53b74eeca7d91b44f01ab94b to finish bootstrapping
I20250812 13:59:11.989374 8170 raft_consensus.cc:1215] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250812 13:59:12.020265 8314 raft_consensus.cc:1215] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250812 13:59:12.130091 8476 raft_consensus.cc:1215] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250812 13:59:12.264874 8294 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 13:59:12.265316 8150 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 13:59:12.269162 8455 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
ecaad71b2ec74b2d806cbaa334eb9ce2 | 127.4.158.62:44775 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.4.158.20:37593 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
2270135b49614f74a5e652804cbf4221 | 127.4.158.2:41823 | HEALTHY | <none> | 1 | 0
73afe7141d5f489d9b82fd2ddf56ceae | 127.4.158.1:35897 | HEALTHY | <none> | 1 | 0
a98794df53b74eeca7d91b44f01ab94b | 127.4.158.3:40737 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.4.158.1 | experimental | 127.4.158.1:35897
local_ip_for_outbound_sockets | 127.4.158.2 | experimental | 127.4.158.2:41823
local_ip_for_outbound_sockets | 127.4.158.3 | experimental | 127.4.158.3:40737
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.4.158.1:35897
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.4.158.2:41823
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.4.158.3:40737
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.4.158.20:37593 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250812 13:59:12.481359 4728 log_verifier.cc:126] Checking tablet 133f2f966d4c454c9aff76354bb58ccd
I20250812 13:59:12.484369 8564 raft_consensus.cc:1062] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: attempting to promote NON_VOTER 73afe7141d5f489d9b82fd2ddf56ceae to VOTER
I20250812 13:59:12.486146 8564 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } }
I20250812 13:59:12.491312 8314 raft_consensus.cc:1273] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Refusing update from remote peer a98794df53b74eeca7d91b44f01ab94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250812 13:59:12.491959 8170 raft_consensus.cc:1273] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEARNER]: Refusing update from remote peer a98794df53b74eeca7d91b44f01ab94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250812 13:59:12.492666 8564 consensus_queue.cc:1035] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250812 13:59:12.493763 8545 consensus_queue.cc:1035] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250812 13:59:12.502754 8545 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } } }
I20250812 13:59:12.504673 8314 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } } }
I20250812 13:59:12.507434 8170 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } } }
I20250812 13:59:12.512584 8564 raft_consensus.cc:1062] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: attempting to promote NON_VOTER 2270135b49614f74a5e652804cbf4221 to VOTER
I20250812 13:59:12.514369 8564 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:12.519790 8025 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 reported cstate change: config changed from index 11 to 12, 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a98794df53b74eeca7d91b44f01ab94b" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: NON_VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: true } } }
I20250812 13:59:12.525359 8170 raft_consensus.cc:1273] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Refusing update from remote peer a98794df53b74eeca7d91b44f01ab94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250812 13:59:12.526211 8314 raft_consensus.cc:1273] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 LEARNER]: Refusing update from remote peer a98794df53b74eeca7d91b44f01ab94b: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250812 13:59:12.526839 8564 consensus_queue.cc:1035] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250812 13:59:12.527789 8562 consensus_queue.cc:1035] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250812 13:59:12.534543 8562 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 2270135b49614f74a5e652804cbf4221 (127.4.158.2) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:12.537382 8170 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 2270135b49614f74a5e652804cbf4221 (127.4.158.2) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:12.539976 8314 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 2270135b49614f74a5e652804cbf4221 (127.4.158.2) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:12.548192 8405 raft_consensus.cc:1062] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: attempting to promote NON_VOTER a98794df53b74eeca7d91b44f01ab94b to VOTER
I20250812 13:59:12.550230 8405 consensus_queue.cc:237] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:12.552224 8026 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae reported cstate change: config changed from index 12 to 13, 2270135b49614f74a5e652804cbf4221 (127.4.158.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a98794df53b74eeca7d91b44f01ab94b" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:12.569823 8476 raft_consensus.cc:1273] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 2 LEARNER]: Refusing update from remote peer 73afe7141d5f489d9b82fd2ddf56ceae: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250812 13:59:12.570412 8314 raft_consensus.cc:1273] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Refusing update from remote peer 73afe7141d5f489d9b82fd2ddf56ceae: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250812 13:59:12.571923 8405 consensus_queue.cc:1035] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250812 13:59:12.573007 8567 consensus_queue.cc:1035] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250812 13:59:12.582747 8567 raft_consensus.cc:2953] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } } }
I20250812 13:59:12.584805 8476 raft_consensus.cc:2953] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } } }
I20250812 13:59:12.585129 8314 raft_consensus.cc:2953] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } } }
I20250812 13:59:12.596414 8027 catalog_manager.cc:5582] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae reported cstate change: config changed from index 11 to 12, a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250812 13:59:12.616703 4728 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 133f2f966d4c454c9aff76354bb58ccd
I20250812 13:59:12.617012 4728 log_verifier.cc:126] Checking tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829
I20250812 13:59:12.699973 4728 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829
I20250812 13:59:12.700209 4728 log_verifier.cc:126] Checking tablet 691e5640a3eb4d208d8f189d92cd044b
I20250812 13:59:12.726792 4728 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 691e5640a3eb4d208d8f189d92cd044b
I20250812 13:59:12.727155 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 7994
I20250812 13:59:12.753635 4728 minidump.cc:252] Setting minidump size limit to 20M
I20250812 13:59:12.754920 4728 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:12.755929 4728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:12.766204 8599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:12.766238 8598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:12.767700 8601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:12.882117 4728 server_base.cc:1047] running on GCE node
I20250812 13:59:12.883256 4728 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250812 13:59:12.883491 4728 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250812 13:59:12.883651 4728 hybrid_clock.cc:648] HybridClock initialized: now 1755007152883634 us; error 0 us; skew 500 ppm
I20250812 13:59:12.884310 4728 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:12.887650 4728 webserver.cc:480] Webserver started at http://0.0.0.0:39271/ using document root <none> and password file <none>
I20250812 13:59:12.888453 4728 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:12.888645 4728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:12.894371 4728 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250812 13:59:12.898099 8606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:12.899019 4728 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250812 13:59:12.899317 4728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2"
format_stamp: "Formatted at 2025-08-12 13:58:51 on dist-test-slave-f6np"
I20250812 13:59:12.900913 4728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:12.918071 4728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:12.919488 4728 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:12.919939 4728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:12.928838 4728 sys_catalog.cc:263] Verifying existing consensus state
W20250812 13:59:12.932461 4728 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.4.158.62:44775 exists but no master address supplied!
I20250812 13:59:12.934522 4728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap starting.
I20250812 13:59:12.975400 4728 log.cc:826] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:13.038909 4728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:13.039695 4728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap complete.
I20250812 13:59:13.053085 4728 raft_consensus.cc:357] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:13.053936 4728 raft_consensus.cc:738] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaad71b2ec74b2d806cbaa334eb9ce2, State: Initialized, Role: FOLLOWER
I20250812 13:59:13.054773 4728 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:13.055248 4728 raft_consensus.cc:397] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:59:13.055508 4728 raft_consensus.cc:491] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:59:13.055822 4728 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 3 FOLLOWER]: Advancing to term 4
I20250812 13:59:13.061339 4728 raft_consensus.cc:513] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:13.062237 4728 leader_election.cc:304] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecaad71b2ec74b2d806cbaa334eb9ce2; no voters:
I20250812 13:59:13.063458 4728 leader_election.cc:290] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250812 13:59:13.063815 8613 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 4 FOLLOWER]: Leader election won for term 4
I20250812 13:59:13.065374 8613 raft_consensus.cc:695] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 4 LEADER]: Becoming Leader. State: Replica: ecaad71b2ec74b2d806cbaa334eb9ce2, State: Running, Role: LEADER
I20250812 13:59:13.067234 8613 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:13.073971 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } } }
I20250812 13:59:13.074200 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecaad71b2ec74b2d806cbaa334eb9ce2. Latest consensus state: current_term: 4 leader_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } } }
I20250812 13:59:13.074779 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: This master's current role is: LEADER
I20250812 13:59:13.074862 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: This master's current role is: LEADER
I20250812 13:59:13.106256 4728 tablet_replica.cc:331] stopping tablet replica
I20250812 13:59:13.106976 4728 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 4 LEADER]: Raft consensus shutting down.
I20250812 13:59:13.107537 4728 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250812 13:59:13.110561 4728 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250812 13:59:13.111207 4728 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250812 13:59:13.139333 4728 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250812 13:59:13.624681 8215 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:44775: connect: Connection refused (error 111)
W20250812 13:59:13.627444 8521 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:44775: connect: Connection refused (error 111)
W20250812 13:59:13.638407 8362 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:44775: connect: Connection refused (error 111)
I20250812 13:59:18.171391 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 8064
I20250812 13:59:18.194383 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 8222
I20250812 13:59:18.218556 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 8366
I20250812 13:59:18.243350 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:44775
--webserver_interface=127.4.158.62
--webserver_port=33517
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:44775 with env {}
W20250812 13:59:18.549280 8690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:18.549952 8690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:18.550436 8690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:18.582406 8690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:59:18.582737 8690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:18.583017 8690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:59:18.583263 8690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:59:18.619724 8690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:44775
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=33517
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:18.621116 8690 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:18.622951 8690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:18.634435 8696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:18.635640 8697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:19.742918 8699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:19.745502 8698 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1106 milliseconds
I20250812 13:59:19.745651 8690 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:19.747210 8690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:19.750197 8690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:19.751613 8690 hybrid_clock.cc:648] HybridClock initialized: now 1755007159751584 us; error 45 us; skew 500 ppm
I20250812 13:59:19.752410 8690 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:19.758857 8690 webserver.cc:480] Webserver started at http://127.4.158.62:33517/ using document root <none> and password file <none>
I20250812 13:59:19.759783 8690 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:19.759990 8690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:19.767786 8690 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250812 13:59:19.773026 8706 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:19.774143 8690 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250812 13:59:19.774479 8690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2"
format_stamp: "Formatted at 2025-08-12 13:58:51 on dist-test-slave-f6np"
I20250812 13:59:19.776425 8690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:19.826987 8690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:19.828480 8690 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:19.828934 8690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:19.897720 8690 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:44775
I20250812 13:59:19.897778 8757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:44775 every 8 connection(s)
I20250812 13:59:19.900611 8690 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:59:19.901294 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 8690
I20250812 13:59:19.903275 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:35897
--local_ip_for_outbound_sockets=127.4.158.1
--tserver_master_addrs=127.4.158.62:44775
--webserver_port=35391
--webserver_interface=127.4.158.1
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:59:19.911793 8758 sys_catalog.cc:263] Verifying existing consensus state
I20250812 13:59:19.916638 8758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap starting.
I20250812 13:59:19.926976 8758 log.cc:826] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:20.003163 8758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:20.003952 8758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Bootstrap complete.
I20250812 13:59:20.023085 8758 raft_consensus.cc:357] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:20.025256 8758 raft_consensus.cc:738] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaad71b2ec74b2d806cbaa334eb9ce2, State: Initialized, Role: FOLLOWER
I20250812 13:59:20.026108 8758 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:20.026652 8758 raft_consensus.cc:397] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:59:20.026903 8758 raft_consensus.cc:491] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:59:20.027205 8758 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 5 FOLLOWER]: Advancing to term 6
I20250812 13:59:20.032423 8758 raft_consensus.cc:513] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:20.033129 8758 leader_election.cc:304] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecaad71b2ec74b2d806cbaa334eb9ce2; no voters:
I20250812 13:59:20.035101 8758 leader_election.cc:290] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250812 13:59:20.035498 8762 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 6 FOLLOWER]: Leader election won for term 6
I20250812 13:59:20.038403 8762 raft_consensus.cc:695] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [term 6 LEADER]: Becoming Leader. State: Replica: ecaad71b2ec74b2d806cbaa334eb9ce2, State: Running, Role: LEADER
I20250812 13:59:20.039279 8762 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } }
I20250812 13:59:20.040721 8758 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:59:20.049172 8764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecaad71b2ec74b2d806cbaa334eb9ce2. Latest consensus state: current_term: 6 leader_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } } }
I20250812 13:59:20.050638 8763 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecaad71b2ec74b2d806cbaa334eb9ce2" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 44775 } } }
I20250812 13:59:20.051107 8764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: This master's current role is: LEADER
I20250812 13:59:20.051301 8763 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2 [sys.catalog]: This master's current role is: LEADER
I20250812 13:59:20.055284 8768 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:59:20.069875 8768 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=3d7f99971cb54e9aba66929ceb029c51]
I20250812 13:59:20.071715 8768 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=a8f9783451d54c3ea15ba376e8416820]
I20250812 13:59:20.073561 8768 catalog_manager.cc:671] Loaded metadata for table TestTable [id=b9c17cb27f7f4d4e832f9ae435d9d7f6]
I20250812 13:59:20.081774 8768 tablet_loader.cc:96] loaded metadata for tablet 133f2f966d4c454c9aff76354bb58ccd (table TestTable [id=b9c17cb27f7f4d4e832f9ae435d9d7f6])
I20250812 13:59:20.088038 8768 tablet_loader.cc:96] loaded metadata for tablet 4bf9ef4b2e5241ca8c8b86a9fe3dc829 (table TestTable1 [id=3d7f99971cb54e9aba66929ceb029c51])
I20250812 13:59:20.089473 8768 tablet_loader.cc:96] loaded metadata for tablet 691e5640a3eb4d208d8f189d92cd044b (table TestTable2 [id=a8f9783451d54c3ea15ba376e8416820])
I20250812 13:59:20.091071 8768 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:59:20.100241 8768 catalog_manager.cc:1261] Loaded cluster ID: 46a83685258d4947931226adb763e652
I20250812 13:59:20.100641 8768 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:59:20.109890 8768 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:59:20.116009 8768 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ecaad71b2ec74b2d806cbaa334eb9ce2: Loaded TSK: 0
I20250812 13:59:20.117698 8768 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250812 13:59:20.238346 8760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:20.238858 8760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:20.239373 8760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:20.271028 8760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:20.272003 8760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 13:59:20.307693 8760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:35897
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=35391
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:20.309137 8760 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:20.310763 8760 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:20.323099 8785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:20.323930 8786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:21.725816 8784 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 8760
W20250812 13:59:21.876030 8784 kernel_stack_watchdog.cc:198] Thread 8760 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 13:59:21.877025 8760 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.552s user 0.000s sys 0.002s
W20250812 13:59:21.877390 8760 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.553s user 0.000s sys 0.002s
W20250812 13:59:21.877432 8787 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1553 milliseconds
W20250812 13:59:21.878883 8789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:21.878960 8760 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:21.882293 8760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:21.884653 8760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:21.886076 8760 hybrid_clock.cc:648] HybridClock initialized: now 1755007161886005 us; error 76 us; skew 500 ppm
I20250812 13:59:21.886837 8760 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:21.893220 8760 webserver.cc:480] Webserver started at http://127.4.158.1:35391/ using document root <none> and password file <none>
I20250812 13:59:21.894148 8760 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:21.894351 8760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:21.902354 8760 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250812 13:59:21.907272 8795 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:21.908390 8760 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250812 13:59:21.908699 8760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73afe7141d5f489d9b82fd2ddf56ceae"
format_stamp: "Formatted at 2025-08-12 13:58:53 on dist-test-slave-f6np"
I20250812 13:59:21.910780 8760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:21.954964 8760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:21.956389 8760 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:21.956825 8760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:21.959736 8760 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:21.965955 8802 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250812 13:59:21.977205 8760 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250812 13:59:21.977418 8760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.001s
I20250812 13:59:21.977682 8760 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250812 13:59:21.982901 8802 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap starting.
I20250812 13:59:21.987380 8760 ts_tablet_manager.cc:610] Registered 2 tablets
I20250812 13:59:21.987658 8760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.009s sys 0.000s
I20250812 13:59:22.053987 8802 log.cc:826] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:22.181736 8760 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:35897
I20250812 13:59:22.181919 8909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:35897 every 8 connection(s)
I20250812 13:59:22.182106 8802 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:22.183185 8802 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap complete.
I20250812 13:59:22.184746 8802 ts_tablet_manager.cc:1397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent bootstrapping tablet: real 0.202s user 0.156s sys 0.040s
I20250812 13:59:22.185184 8760 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 13:59:22.190629 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 8760
I20250812 13:59:22.192461 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:41823
--local_ip_for_outbound_sockets=127.4.158.2
--tserver_master_addrs=127.4.158.62:44775
--webserver_port=37581
--webserver_interface=127.4.158.2
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:59:22.207365 8802 raft_consensus.cc:357] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:22.211300 8802 raft_consensus.cc:738] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73afe7141d5f489d9b82fd2ddf56ceae, State: Initialized, Role: FOLLOWER
I20250812 13:59:22.212522 8802 consensus_queue.cc:260] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:22.219552 8802 ts_tablet_manager.cc:1428] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent starting tablet: real 0.034s user 0.033s sys 0.000s
I20250812 13:59:22.220158 8802 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap starting.
I20250812 13:59:22.242923 8910 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:59:22.243606 8910 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:22.245244 8910 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:59:22.251883 8723 ts_manager.cc:194] Registered new tserver with Master: 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897)
I20250812 13:59:22.257601 8723 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 2270135b49614f74a5e652804cbf4221 (127.4.158.2) added, VOTER 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) added, VOTER a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:22.325404 8723 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:48977
I20250812 13:59:22.330199 8910 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
I20250812 13:59:22.387871 8802 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:22.388782 8802 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Bootstrap complete.
I20250812 13:59:22.390331 8802 ts_tablet_manager.cc:1397] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent bootstrapping tablet: real 0.170s user 0.142s sys 0.023s
I20250812 13:59:22.392537 8802 raft_consensus.cc:357] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:22.393157 8802 raft_consensus.cc:738] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73afe7141d5f489d9b82fd2ddf56ceae, State: Initialized, Role: FOLLOWER
I20250812 13:59:22.393764 8802 consensus_queue.cc:260] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:22.395658 8802 ts_tablet_manager.cc:1428] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250812 13:59:22.572093 8914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:22.572598 8914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:22.573089 8914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:22.604420 8914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:22.605268 8914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 13:59:22.640358 8914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:41823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=37581
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:22.641677 8914 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:22.643525 8914 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:22.655165 8926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:22.655755 8927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:22.658897 8929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:23.754554 8928 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1093 milliseconds
I20250812 13:59:23.754669 8914 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:23.755872 8914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:23.758414 8914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:23.759814 8914 hybrid_clock.cc:648] HybridClock initialized: now 1755007163759780 us; error 55 us; skew 500 ppm
I20250812 13:59:23.760607 8914 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:23.766769 8914 webserver.cc:480] Webserver started at http://127.4.158.2:37581/ using document root <none> and password file <none>
I20250812 13:59:23.767670 8914 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:23.767871 8914 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:23.775518 8914 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250812 13:59:23.780349 8936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:23.781387 8914 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250812 13:59:23.781702 8914 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2270135b49614f74a5e652804cbf4221"
format_stamp: "Formatted at 2025-08-12 13:58:55 on dist-test-slave-f6np"
I20250812 13:59:23.783761 8914 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:23.828158 8914 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:23.829610 8914 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:23.830067 8914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:23.832502 8914 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:23.838634 8943 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250812 13:59:23.853976 8914 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250812 13:59:23.854255 8914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.002s
I20250812 13:59:23.854524 8914 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250812 13:59:23.859838 8943 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Bootstrap starting.
I20250812 13:59:23.865826 8914 ts_tablet_manager.cc:610] Registered 3 tablets
I20250812 13:59:23.866110 8914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.011s sys 0.001s
I20250812 13:59:23.915922 8943 log.cc:826] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:23.997599 9016 raft_consensus.cc:491] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:59:23.998457 9016 raft_consensus.cc:513] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:24.001649 9026 raft_consensus.cc:491] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:59:24.002252 9026 raft_consensus.cc:513] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:24.009191 9016 leader_election.cc:290] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737), 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823)
I20250812 13:59:24.009236 9026 leader_election.cc:290] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823), a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737)
W20250812 13:59:24.017665 8799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111)
W20250812 13:59:24.024107 8799 leader_election.cc:336] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111)
W20250812 13:59:24.024735 8798 leader_election.cc:336] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823): Network error: Client connection negotiation failed: client connection to 127.4.158.2:41823: connect: Connection refused (error 111)
W20250812 13:59:24.027381 8798 leader_election.cc:336] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823): Network error: Client connection negotiation failed: client connection to 127.4.158.2:41823: connect: Connection refused (error 111)
W20250812 13:59:24.027441 8799 leader_election.cc:336] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111)
I20250812 13:59:24.027931 8798 leader_election.cc:304] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73afe7141d5f489d9b82fd2ddf56ceae; no voters: 2270135b49614f74a5e652804cbf4221, a98794df53b74eeca7d91b44f01ab94b
I20250812 13:59:24.028062 8799 leader_election.cc:304] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73afe7141d5f489d9b82fd2ddf56ceae; no voters: 2270135b49614f74a5e652804cbf4221, a98794df53b74eeca7d91b44f01ab94b
I20250812 13:59:24.028834 9026 raft_consensus.cc:2747] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250812 13:59:24.028909 9016 raft_consensus.cc:2747] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250812 13:59:24.029580 8943 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:24.030438 8943 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Bootstrap complete.
I20250812 13:59:24.031482 8943 ts_tablet_manager.cc:1397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Time spent bootstrapping tablet: real 0.172s user 0.133s sys 0.032s
I20250812 13:59:24.045760 8943 raft_consensus.cc:357] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:24.047927 8943 raft_consensus.cc:738] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Initialized, Role: FOLLOWER
I20250812 13:59:24.048780 8943 consensus_queue.cc:260] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:24.053185 8943 ts_tablet_manager.cc:1428] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221: Time spent starting tablet: real 0.021s user 0.019s sys 0.002s
I20250812 13:59:24.053917 8943 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Bootstrap starting.
I20250812 13:59:24.067030 8914 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:41823
I20250812 13:59:24.067216 9056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:41823 every 8 connection(s)
I20250812 13:59:24.070533 8914 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 13:59:24.072036 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 8914
I20250812 13:59:24.073751 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:40737
--local_ip_for_outbound_sockets=127.4.158.3
--tserver_master_addrs=127.4.158.62:44775
--webserver_port=43395
--webserver_interface=127.4.158.3
--builtin_ntp_servers=127.4.158.20:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250812 13:59:24.099138 9057 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:59:24.099654 9057 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:24.100948 9057 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:59:24.105445 8722 ts_manager.cc:194] Registered new tserver with Master: 2270135b49614f74a5e652804cbf4221 (127.4.158.2:41823)
I20250812 13:59:24.110690 8722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:45251
I20250812 13:59:24.171907 8943 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:24.172585 8943 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Bootstrap complete.
I20250812 13:59:24.173552 8943 ts_tablet_manager.cc:1397] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Time spent bootstrapping tablet: real 0.120s user 0.093s sys 0.020s
I20250812 13:59:24.175143 8943 raft_consensus.cc:357] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:24.175590 8943 raft_consensus.cc:738] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Initialized, Role: FOLLOWER
I20250812 13:59:24.176039 8943 consensus_queue.cc:260] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:24.176985 9057 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
I20250812 13:59:24.177376 8943 ts_tablet_manager.cc:1428] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250812 13:59:24.177922 8943 tablet_bootstrap.cc:492] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Bootstrap starting.
I20250812 13:59:24.272415 8943 tablet_bootstrap.cc:492] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:24.273083 8943 tablet_bootstrap.cc:492] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Bootstrap complete.
I20250812 13:59:24.274060 8943 ts_tablet_manager.cc:1397] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Time spent bootstrapping tablet: real 0.096s user 0.084s sys 0.011s
I20250812 13:59:24.275518 8943 raft_consensus.cc:357] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:59:24.275882 8943 raft_consensus.cc:738] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Initialized, Role: FOLLOWER
I20250812 13:59:24.276340 8943 consensus_queue.cc:260] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:59:24.276748 8943 raft_consensus.cc:397] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:59:24.277011 8943 raft_consensus.cc:491] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:59:24.277302 8943 raft_consensus.cc:3058] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Advancing to term 3
I20250812 13:59:24.282626 8943 raft_consensus.cc:513] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:59:24.283254 8943 leader_election.cc:304] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2270135b49614f74a5e652804cbf4221; no voters:
I20250812 13:59:24.283738 8943 leader_election.cc:290] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250812 13:59:24.283972 9045 raft_consensus.cc:2802] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 3 FOLLOWER]: Leader election won for term 3
I20250812 13:59:24.286928 8943 ts_tablet_manager.cc:1428] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20250812 13:59:24.287385 9045 raft_consensus.cc:695] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [term 3 LEADER]: Becoming Leader. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Running, Role: LEADER
I20250812 13:59:24.288163 9045 consensus_queue.cc:237] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } }
I20250812 13:59:24.300385 8722 catalog_manager.cc:5582] T 691e5640a3eb4d208d8f189d92cd044b P 2270135b49614f74a5e652804cbf4221 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "2270135b49614f74a5e652804cbf4221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } health_report { overall_health: HEALTHY } } }
W20250812 13:59:24.395565 9061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:24.396136 9061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:24.396652 9061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:24.429500 9061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:24.430409 9061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 13:59:24.465724 9061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:40737
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=43395
--tserver_master_addrs=127.4.158.62:44775
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:24.467139 9061 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:24.468741 9061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:24.480302 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:25.349073 9083 raft_consensus.cc:491] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:59:25.349860 9083 raft_consensus.cc:513] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:25.364300 9083 leader_election.cc:290] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737), 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897)
W20250812 13:59:25.367131 8940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111)
W20250812 13:59:25.377839 8940 leader_election.cc:336] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111)
I20250812 13:59:25.378845 8865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" candidate_uuid: "2270135b49614f74a5e652804cbf4221" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" is_pre_election: true
I20250812 13:59:25.379703 8865 raft_consensus.cc:2466] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2270135b49614f74a5e652804cbf4221 in term 2.
I20250812 13:59:25.381326 8937 leader_election.cc:304] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2270135b49614f74a5e652804cbf4221, 73afe7141d5f489d9b82fd2ddf56ceae; no voters: a98794df53b74eeca7d91b44f01ab94b
I20250812 13:59:25.382359 9083 raft_consensus.cc:2802] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250812 13:59:25.382710 9083 raft_consensus.cc:491] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:59:25.383008 9083 raft_consensus.cc:3058] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Advancing to term 3
I20250812 13:59:25.390024 9083 raft_consensus.cc:513] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
W20250812 13:59:25.396718 8940 leader_election.cc:336] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111)
I20250812 13:59:25.399021 8865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" candidate_uuid: "2270135b49614f74a5e652804cbf4221" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "73afe7141d5f489d9b82fd2ddf56ceae"
I20250812 13:59:25.399657 8865 raft_consensus.cc:3058] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Advancing to term 3
I20250812 13:59:25.409969 9083 leader_election.cc:290] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 election: Requested vote from peers a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737), 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897)
I20250812 13:59:25.409650 8865 raft_consensus.cc:2466] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2270135b49614f74a5e652804cbf4221 in term 3.
I20250812 13:59:25.411844 8937 leader_election.cc:304] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2270135b49614f74a5e652804cbf4221, 73afe7141d5f489d9b82fd2ddf56ceae; no voters: a98794df53b74eeca7d91b44f01ab94b
I20250812 13:59:25.413081 9083 raft_consensus.cc:2802] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 3 FOLLOWER]: Leader election won for term 3
I20250812 13:59:25.424023 9083 raft_consensus.cc:695] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 3 LEADER]: Becoming Leader. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Running, Role: LEADER
I20250812 13:59:25.425041 9083 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:25.438675 8722 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 reported cstate change: term changed from 2 to 3, leader changed from <none> to 2270135b49614f74a5e652804cbf4221 (127.4.158.2). New cstate: current_term: 3 leader_uuid: "2270135b49614f74a5e652804cbf4221" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250812 13:59:25.797948 9083 raft_consensus.cc:491] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:59:25.798491 9083 raft_consensus.cc:513] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:25.800491 9083 leader_election.cc:290] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897), a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737)
I20250812 13:59:25.804739 8865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bf9ef4b2e5241ca8c8b86a9fe3dc829" candidate_uuid: "2270135b49614f74a5e652804cbf4221" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" is_pre_election: true
I20250812 13:59:25.805413 8865 raft_consensus.cc:2466] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2270135b49614f74a5e652804cbf4221 in term 2.
I20250812 13:59:25.807089 8937 leader_election.cc:304] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2270135b49614f74a5e652804cbf4221, 73afe7141d5f489d9b82fd2ddf56ceae; no voters:
W20250812 13:59:25.811511 8940 leader_election.cc:336] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111)
I20250812 13:59:25.812117 9083 raft_consensus.cc:2802] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250812 13:59:25.813053 9083 raft_consensus.cc:491] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:59:25.813674 9083 raft_consensus.cc:3058] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 2 FOLLOWER]: Advancing to term 3
I20250812 13:59:25.820684 9083 raft_consensus.cc:513] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:25.823918 8865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4bf9ef4b2e5241ca8c8b86a9fe3dc829" candidate_uuid: "2270135b49614f74a5e652804cbf4221" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "73afe7141d5f489d9b82fd2ddf56ceae"
I20250812 13:59:25.824517 8865 raft_consensus.cc:3058] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 2 FOLLOWER]: Advancing to term 3
I20250812 13:59:25.831631 8865 raft_consensus.cc:2466] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2270135b49614f74a5e652804cbf4221 in term 3.
I20250812 13:59:25.833014 8937 leader_election.cc:304] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2270135b49614f74a5e652804cbf4221, 73afe7141d5f489d9b82fd2ddf56ceae; no voters:
I20250812 13:59:25.834122 9088 raft_consensus.cc:2802] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 3 FOLLOWER]: Leader election won for term 3
W20250812 13:59:25.846115 8940 leader_election.cc:336] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111)
I20250812 13:59:25.847479 9083 leader_election.cc:290] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [CANDIDATE]: Term 3 election: Requested vote from peers 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897), a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737)
I20250812 13:59:25.852702 9088 raft_consensus.cc:695] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [term 3 LEADER]: Becoming Leader. State: Replica: 2270135b49614f74a5e652804cbf4221, State: Running, Role: LEADER
I20250812 13:59:25.853691 9088 consensus_queue.cc:237] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:25.869903 8722 catalog_manager.cc:5582] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 reported cstate change: term changed from 2 to 3, leader changed from 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) to 2270135b49614f74a5e652804cbf4221 (127.4.158.2). New cstate: current_term: 3 leader_uuid: "2270135b49614f74a5e652804cbf4221" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250812 13:59:25.875511 8865 raft_consensus.cc:1273] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 3 FOLLOWER]: Refusing update from remote peer 2270135b49614f74a5e652804cbf4221: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250812 13:59:25.878767 9094 consensus_queue.cc:1035] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20250812 13:59:25.938171 8940 consensus_peers.cc:489] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 -> Peer a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): Couldn't send request to peer a98794df53b74eeca7d91b44f01ab94b. Status: Network error: Client connection negotiation failed: client connection to 127.4.158.3:40737: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250812 13:59:25.883428 9076 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 9061
W20250812 13:59:25.981499 9061 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.500s user 0.614s sys 0.875s
W20250812 13:59:25.981886 9061 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.501s user 0.614s sys 0.875s
W20250812 13:59:24.481487 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:25.983865 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:25.986493 9079 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1501 milliseconds
I20250812 13:59:25.986518 9061 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:25.987712 9061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:25.990173 9061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:25.991699 9061 hybrid_clock.cc:648] HybridClock initialized: now 1755007165991615 us; error 66 us; skew 500 ppm
I20250812 13:59:25.992729 9061 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:25.995332 9006 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:25.999619 8864 raft_consensus.cc:1273] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 3 FOLLOWER]: Refusing update from remote peer 2270135b49614f74a5e652804cbf4221: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250812 13:59:26.000231 9061 webserver.cc:480] Webserver started at http://127.4.158.3:43395/ using document root <none> and password file <none>
I20250812 13:59:26.000783 9088 consensus_queue.cc:1035] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250812 13:59:26.001194 9061 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:26.001428 9061 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:26.006203 9083 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:26.007500 8864 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:26.016116 9061 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.006s sys 0.006s
I20250812 13:59:26.017622 8722 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae reported cstate change: config changed from index 13 to 15, VOTER a98794df53b74eeca7d91b44f01ab94b (127.4.158.3) evicted. New cstate: current_term: 3 leader_uuid: "2270135b49614f74a5e652804cbf4221" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:26.018682 8709 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 133f2f966d4c454c9aff76354bb58ccd with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250812 13:59:26.021726 9106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:26.023056 9061 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
W20250812 13:59:26.025139 8722 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 133f2f966d4c454c9aff76354bb58ccd on TS a98794df53b74eeca7d91b44f01ab94b: Not found: failed to reset TS proxy: Could not find TS for UUID a98794df53b74eeca7d91b44f01ab94b
I20250812 13:59:26.023401 9061 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a98794df53b74eeca7d91b44f01ab94b"
format_stamp: "Formatted at 2025-08-12 13:58:57 on dist-test-slave-f6np"
I20250812 13:59:26.026013 9061 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:26.029078 9006 consensus_queue.cc:237] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:26.031025 9083 raft_consensus.cc:2953] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } }
I20250812 13:59:26.037056 8709 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 133f2f966d4c454c9aff76354bb58ccd with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250812 13:59:26.040190 8723 catalog_manager.cc:5582] T 133f2f966d4c454c9aff76354bb58ccd P 2270135b49614f74a5e652804cbf4221 reported cstate change: config changed from index 15 to 16, VOTER 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1) evicted. New cstate: current_term: 3 leader_uuid: "2270135b49614f74a5e652804cbf4221" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250812 13:59:26.056855 8845 tablet_service.cc:1515] Processing DeleteTablet for tablet 133f2f966d4c454c9aff76354bb58ccd with delete_type TABLET_DATA_TOMBSTONED (TS 73afe7141d5f489d9b82fd2ddf56ceae not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:50688
I20250812 13:59:26.059469 9108 tablet_replica.cc:331] stopping tablet replica
I20250812 13:59:26.060113 9108 raft_consensus.cc:2241] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 3 FOLLOWER]: Raft consensus shutting down.
I20250812 13:59:26.060602 9108 raft_consensus.cc:2270] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae [term 3 FOLLOWER]: Raft consensus is shut down!
I20250812 13:59:26.063603 9108 ts_tablet_manager.cc:1905] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20250812 13:59:26.077625 8708 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 133f2f966d4c454c9aff76354bb58ccd on TS a98794df53b74eeca7d91b44f01ab94b failed: Not found: failed to reset TS proxy: Could not find TS for UUID a98794df53b74eeca7d91b44f01ab94b
I20250812 13:59:26.079878 9108 ts_tablet_manager.cc:1918] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250812 13:59:26.080304 9108 log.cc:1199] T 133f2f966d4c454c9aff76354bb58ccd P 73afe7141d5f489d9b82fd2ddf56ceae: Deleting WAL directory at /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/wals/133f2f966d4c454c9aff76354bb58ccd
I20250812 13:59:26.082079 8707 catalog_manager.cc:4928] TS 73afe7141d5f489d9b82fd2ddf56ceae (127.4.158.1:35897): tablet 133f2f966d4c454c9aff76354bb58ccd (table TestTable [id=b9c17cb27f7f4d4e832f9ae435d9d7f6]) successfully deleted
I20250812 13:59:26.100477 9061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:26.102066 9061 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:26.102530 9061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:26.105048 9061 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:26.111104 9115 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250812 13:59:26.122509 9061 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250812 13:59:26.122749 9061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.002s
I20250812 13:59:26.123114 9061 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250812 13:59:26.128433 9115 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Bootstrap starting.
I20250812 13:59:26.131032 9061 ts_tablet_manager.cc:610] Registered 2 tablets
I20250812 13:59:26.131239 9061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.003s sys 0.004s
I20250812 13:59:26.192549 9115 log.cc:826] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:26.294342 9061 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:40737
I20250812 13:59:26.294416 9222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:40737 every 8 connection(s)
I20250812 13:59:26.297874 9061 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 13:59:26.300817 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 9061
I20250812 13:59:26.320528 9115 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:26.321288 9115 tablet_bootstrap.cc:492] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Bootstrap complete.
I20250812 13:59:26.322594 9115 ts_tablet_manager.cc:1397] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Time spent bootstrapping tablet: real 0.195s user 0.153s sys 0.036s
I20250812 13:59:26.334619 9223 heartbeater.cc:344] Connected to a master server at 127.4.158.62:44775
I20250812 13:59:26.335237 9223 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:26.336340 9223 heartbeater.cc:507] Master 127.4.158.62:44775 requested a full tablet report, sending...
I20250812 13:59:26.340266 8722 ts_manager.cc:194] Registered new tserver with Master: a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737)
I20250812 13:59:26.339329 9115 raft_consensus.cc:357] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:26.342620 9115 raft_consensus.cc:738] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98794df53b74eeca7d91b44f01ab94b, State: Initialized, Role: FOLLOWER
I20250812 13:59:26.343457 9115 consensus_queue.cc:260] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } } peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false } }
I20250812 13:59:26.344521 8722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:60213
I20250812 13:59:26.347505 9115 ts_tablet_manager.cc:1428] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b: Time spent starting tablet: real 0.025s user 0.020s sys 0.004s
I20250812 13:59:26.348248 9115 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Bootstrap starting.
I20250812 13:59:26.349027 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 13:59:26.349485 9223 heartbeater.cc:499] Master 127.4.158.62:44775 was elected leader, sending a full tablet report...
I20250812 13:59:26.349822 9178 raft_consensus.cc:3058] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Advancing to term 3
I20250812 13:59:26.353250 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250812 13:59:26.356304 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250812 13:59:26.359318 9178 raft_consensus.cc:1273] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P a98794df53b74eeca7d91b44f01ab94b [term 3 FOLLOWER]: Refusing update from remote peer 2270135b49614f74a5e652804cbf4221: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250812 13:59:26.361312 9083 consensus_queue.cc:1035] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250812 13:59:26.373237 8864 raft_consensus.cc:1273] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 73afe7141d5f489d9b82fd2ddf56ceae [term 3 FOLLOWER]: Refusing update from remote peer 2270135b49614f74a5e652804cbf4221: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250812 13:59:26.374665 9088 consensus_queue.cc:1035] T 4bf9ef4b2e5241ca8c8b86a9fe3dc829 P 2270135b49614f74a5e652804cbf4221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250812 13:59:26.405524 9158 tablet_service.cc:1515] Processing DeleteTablet for tablet 133f2f966d4c454c9aff76354bb58ccd with delete_type TABLET_DATA_TOMBSTONED (TS a98794df53b74eeca7d91b44f01ab94b not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:42162
W20250812 13:59:26.411479 8710 catalog_manager.cc:4908] TS a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): delete failed for tablet 133f2f966d4c454c9aff76354bb58ccd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 133f2f966d4c454c9aff76354bb58ccd already in progress: opening tablet
I20250812 13:59:26.464795 9115 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 13:59:26.465488 9115 tablet_bootstrap.cc:492] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Bootstrap complete.
I20250812 13:59:26.466477 9115 ts_tablet_manager.cc:1397] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Time spent bootstrapping tablet: real 0.118s user 0.101s sys 0.015s
I20250812 13:59:26.468017 9115 raft_consensus.cc:357] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:26.468444 9115 raft_consensus.cc:738] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98794df53b74eeca7d91b44f01ab94b, State: Initialized, Role: FOLLOWER
I20250812 13:59:26.468926 9115 consensus_queue.cc:260] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "a98794df53b74eeca7d91b44f01ab94b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 40737 } } peers { permanent_uuid: "73afe7141d5f489d9b82fd2ddf56ceae" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35897 } attrs { promote: false } } peers { permanent_uuid: "2270135b49614f74a5e652804cbf4221" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 41823 } attrs { promote: false } }
I20250812 13:59:26.470175 9115 ts_tablet_manager.cc:1428] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250812 13:59:26.474649 9158 tablet_service.cc:1515] Processing DeleteTablet for tablet 133f2f966d4c454c9aff76354bb58ccd with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 13 (current committed config index is 16)) from {username='slave'} at 127.0.0.1:42162
I20250812 13:59:26.475553 9235 tablet_replica.cc:331] stopping tablet replica
I20250812 13:59:26.476270 9235 raft_consensus.cc:2241] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250812 13:59:26.476799 9235 raft_consensus.cc:2270] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250812 13:59:26.479763 9235 ts_tablet_manager.cc:1905] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250812 13:59:26.489444 9235 ts_tablet_manager.cc:1918] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250812 13:59:26.489738 9235 log.cc:1199] T 133f2f966d4c454c9aff76354bb58ccd P a98794df53b74eeca7d91b44f01ab94b: Deleting WAL directory at /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/wals/133f2f966d4c454c9aff76354bb58ccd
I20250812 13:59:26.491235 8710 catalog_manager.cc:4928] TS a98794df53b74eeca7d91b44f01ab94b (127.4.158.3:40737): tablet 133f2f966d4c454c9aff76354bb58ccd (table TestTable [id=b9c17cb27f7f4d4e832f9ae435d9d7f6]) successfully deleted
W20250812 13:59:27.360536 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:28.364636 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:29.368443 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:30.371956 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:31.375540 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:32.378916 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:33.382575 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:34.386083 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:35.389623 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:36.393229 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:37.397274 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:38.401158 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:39.405076 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:40.408718 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:41.412082 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:42.415488 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:43.418859 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:44.422041 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250812 13:59:45.425432 4728 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 133f2f966d4c454c9aff76354bb58ccd: tablet_id: "133f2f966d4c454c9aff76354bb58ccd" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250812 13:59:46.428660 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 8760
I20250812 13:59:46.450050 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 8914
I20250812 13:59:46.474699 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 9061
I20250812 13:59:46.495625 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 8690
2025-08-12T13:59:46Z chronyd exiting
I20250812 13:59:46.541879 4728 test_util.cc:183] -----------------------------------------------
I20250812 13:59:46.542085 4728 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755007066725366-4728-0
[ FAILED ] AdminCliTest.TestRebuildTables (56320 ms)
[----------] 5 tests from AdminCliTest (119750 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250812 13:59:46.545903 4728 test_util.cc:276] Using random seed: 1945021654
I20250812 13:59:46.550102 4728 ts_itest-base.cc:115] Starting cluster with:
I20250812 13:59:46.550254 4728 ts_itest-base.cc:116] --------------
I20250812 13:59:46.550405 4728 ts_itest-base.cc:117] 5 tablet servers
I20250812 13:59:46.550532 4728 ts_itest-base.cc:118] 3 replicas per TS
I20250812 13:59:46.550660 4728 ts_itest-base.cc:119] --------------
2025-08-12T13:59:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T13:59:46Z Disabled control of system clock
I20250812 13:59:46.590539 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:38801
--webserver_interface=127.4.158.62
--webserver_port=0
--builtin_ntp_servers=127.4.158.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:38801
--raft_prepare_replacement_before_eviction=true with env {}
W20250812 13:59:46.884126 9257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:46.884722 9257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:46.885172 9257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:46.916070 9257 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 13:59:46.916518 9257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 13:59:46.916873 9257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:46.917197 9257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 13:59:46.917466 9257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 13:59:46.953277 9257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:38801
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:46.954654 9257 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:46.956243 9257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:46.966663 9263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:46.967123 9264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:46.971061 9266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:48.053717 9265 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250812 13:59:48.053772 9257 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:48.057808 9257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:48.060776 9257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:48.062175 9257 hybrid_clock.cc:648] HybridClock initialized: now 1755007188062122 us; error 73 us; skew 500 ppm
I20250812 13:59:48.062929 9257 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:48.069101 9257 webserver.cc:480] Webserver started at http://127.4.158.62:37963/ using document root <none> and password file <none>
I20250812 13:59:48.070010 9257 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:48.070200 9257 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:48.070606 9257 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:59:48.074882 9257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3533f33f634149419bf8695a046864a9"
format_stamp: "Formatted at 2025-08-12 13:59:48 on dist-test-slave-f6np"
I20250812 13:59:48.075888 9257 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3533f33f634149419bf8695a046864a9"
format_stamp: "Formatted at 2025-08-12 13:59:48 on dist-test-slave-f6np"
I20250812 13:59:48.082719 9257 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250812 13:59:48.087934 9274 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:48.088877 9257 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250812 13:59:48.089149 9257 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3533f33f634149419bf8695a046864a9"
format_stamp: "Formatted at 2025-08-12 13:59:48 on dist-test-slave-f6np"
I20250812 13:59:48.089437 9257 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:48.134600 9257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:48.135973 9257 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:48.136353 9257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:48.202863 9257 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:38801
I20250812 13:59:48.202939 9325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:38801 every 8 connection(s)
I20250812 13:59:48.205451 9257 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 13:59:48.210317 9326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:59:48.213032 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 9257
I20250812 13:59:48.213660 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 13:59:48.232685 9326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9: Bootstrap starting.
I20250812 13:59:48.238924 9326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9: Neither blocks nor log segments found. Creating new log.
I20250812 13:59:48.240720 9326 log.cc:826] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:48.245057 9326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9: No bootstrap required, opened a new log
I20250812 13:59:48.262307 9326 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3533f33f634149419bf8695a046864a9" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 38801 } }
I20250812 13:59:48.262949 9326 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:59:48.263185 9326 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3533f33f634149419bf8695a046864a9, State: Initialized, Role: FOLLOWER
I20250812 13:59:48.263814 9326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3533f33f634149419bf8695a046864a9" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 38801 } }
I20250812 13:59:48.264309 9326 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 13:59:48.264564 9326 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 13:59:48.264865 9326 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:59:48.268743 9326 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3533f33f634149419bf8695a046864a9" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 38801 } }
I20250812 13:59:48.269474 9326 leader_election.cc:304] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3533f33f634149419bf8695a046864a9; no voters:
I20250812 13:59:48.271299 9326 leader_election.cc:290] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 13:59:48.271950 9331 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:59:48.273905 9331 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [term 1 LEADER]: Becoming Leader. State: Replica: 3533f33f634149419bf8695a046864a9, State: Running, Role: LEADER
I20250812 13:59:48.274672 9331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3533f33f634149419bf8695a046864a9" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 38801 } }
I20250812 13:59:48.275609 9326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 13:59:48.285638 9333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3533f33f634149419bf8695a046864a9. Latest consensus state: current_term: 1 leader_uuid: "3533f33f634149419bf8695a046864a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3533f33f634149419bf8695a046864a9" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 38801 } } }
I20250812 13:59:48.286460 9333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [sys.catalog]: This master's current role is: LEADER
I20250812 13:59:48.285614 9332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3533f33f634149419bf8695a046864a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3533f33f634149419bf8695a046864a9" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 38801 } } }
I20250812 13:59:48.287878 9332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9 [sys.catalog]: This master's current role is: LEADER
I20250812 13:59:48.294411 9340 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 13:59:48.304318 9340 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 13:59:48.321427 9340 catalog_manager.cc:1349] Generated new cluster ID: f649435c712a4cdf8e9c135af140e4e0
I20250812 13:59:48.321743 9340 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 13:59:48.335450 9340 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 13:59:48.336808 9340 catalog_manager.cc:1506] Loading token signing keys...
I20250812 13:59:48.360812 9340 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3533f33f634149419bf8695a046864a9: Generated new TSK 0
I20250812 13:59:48.361918 9340 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 13:59:48.388432 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:0
--local_ip_for_outbound_sockets=127.4.158.1
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--builtin_ntp_servers=127.4.158.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250812 13:59:48.688001 9350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:48.688498 9350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:48.688978 9350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:48.720405 9350 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 13:59:48.720796 9350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:48.721581 9350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 13:59:48.757149 9350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:48.758489 9350 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:48.760007 9350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:48.772758 9356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:48.773689 9357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:50.173342 9355 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 9350
W20250812 13:59:50.246273 9355 kernel_stack_watchdog.cc:198] Thread 9350 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 13:59:50.247498 9350 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.475s user 0.518s sys 0.805s
W20250812 13:59:50.247865 9350 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.476s user 0.518s sys 0.805s
W20250812 13:59:50.248497 9359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:50.251684 9358 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1473 milliseconds
I20250812 13:59:50.251686 9350 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:50.252943 9350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:50.255146 9350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:50.256493 9350 hybrid_clock.cc:648] HybridClock initialized: now 1755007190256458 us; error 46 us; skew 500 ppm
I20250812 13:59:50.257248 9350 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:50.263468 9350 webserver.cc:480] Webserver started at http://127.4.158.1:34811/ using document root <none> and password file <none>
I20250812 13:59:50.264360 9350 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:50.264622 9350 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:50.265053 9350 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:59:50.269320 9350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "064ab3024a45485a8299fc7188195546"
format_stamp: "Formatted at 2025-08-12 13:59:50 on dist-test-slave-f6np"
I20250812 13:59:50.270434 9350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "064ab3024a45485a8299fc7188195546"
format_stamp: "Formatted at 2025-08-12 13:59:50 on dist-test-slave-f6np"
I20250812 13:59:50.278128 9350 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250812 13:59:50.283842 9366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:50.284858 9350 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250812 13:59:50.285178 9350 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "064ab3024a45485a8299fc7188195546"
format_stamp: "Formatted at 2025-08-12 13:59:50 on dist-test-slave-f6np"
I20250812 13:59:50.285486 9350 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:50.332257 9350 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:50.333676 9350 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:50.334112 9350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:50.336911 9350 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:50.341313 9350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:59:50.341511 9350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:50.341763 9350 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:59:50.341943 9350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250812 13:59:50.497136 9350 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:34829
I20250812 13:59:50.497294 9478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:34829 every 8 connection(s)
I20250812 13:59:50.499698 9350 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 13:59:50.510495 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 9350
I20250812 13:59:50.511016 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 13:59:50.518110 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:0
--local_ip_for_outbound_sockets=127.4.158.2
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--builtin_ntp_servers=127.4.158.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250812 13:59:50.526290 9479 heartbeater.cc:344] Connected to a master server at 127.4.158.62:38801
I20250812 13:59:50.526803 9479 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:50.528072 9479 heartbeater.cc:507] Master 127.4.158.62:38801 requested a full tablet report, sending...
I20250812 13:59:50.531075 9291 ts_manager.cc:194] Registered new tserver with Master: 064ab3024a45485a8299fc7188195546 (127.4.158.1:34829)
I20250812 13:59:50.533795 9291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:58939
W20250812 13:59:50.812788 9483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:50.813288 9483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:50.813787 9483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:50.844779 9483 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 13:59:50.845171 9483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:50.845981 9483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 13:59:50.882198 9483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:50.883487 9483 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:50.885035 9483 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:50.896334 9489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:51.537904 9479 heartbeater.cc:499] Master 127.4.158.62:38801 was elected leader, sending a full tablet report...
W20250812 13:59:50.897593 9490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:52.017305 9492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:52.019483 9491 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1118 milliseconds
I20250812 13:59:52.019598 9483 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:52.020877 9483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:52.023533 9483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:52.024963 9483 hybrid_clock.cc:648] HybridClock initialized: now 1755007192024922 us; error 53 us; skew 500 ppm
I20250812 13:59:52.025775 9483 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:52.032212 9483 webserver.cc:480] Webserver started at http://127.4.158.2:38513/ using document root <none> and password file <none>
I20250812 13:59:52.033129 9483 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:52.033330 9483 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:52.033818 9483 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:59:52.038426 9483 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4191670a029c46dcab2f80a44580cb2e"
format_stamp: "Formatted at 2025-08-12 13:59:52 on dist-test-slave-f6np"
I20250812 13:59:52.039510 9483 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4191670a029c46dcab2f80a44580cb2e"
format_stamp: "Formatted at 2025-08-12 13:59:52 on dist-test-slave-f6np"
I20250812 13:59:52.046386 9483 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250812 13:59:52.051928 9499 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:52.052910 9483 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250812 13:59:52.053200 9483 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4191670a029c46dcab2f80a44580cb2e"
format_stamp: "Formatted at 2025-08-12 13:59:52 on dist-test-slave-f6np"
I20250812 13:59:52.053501 9483 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:52.110404 9483 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:52.111824 9483 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:52.112236 9483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:52.114687 9483 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:52.118681 9483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:59:52.118873 9483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:52.119103 9483 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:59:52.119247 9483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:52.245309 9483 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:45969
I20250812 13:59:52.245496 9611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:45969 every 8 connection(s)
I20250812 13:59:52.247851 9483 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250812 13:59:52.251130 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 9483
I20250812 13:59:52.251652 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250812 13:59:52.258822 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:0
--local_ip_for_outbound_sockets=127.4.158.3
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--builtin_ntp_servers=127.4.158.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250812 13:59:52.268961 9612 heartbeater.cc:344] Connected to a master server at 127.4.158.62:38801
I20250812 13:59:52.269474 9612 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:52.270768 9612 heartbeater.cc:507] Master 127.4.158.62:38801 requested a full tablet report, sending...
I20250812 13:59:52.273017 9291 ts_manager.cc:194] Registered new tserver with Master: 4191670a029c46dcab2f80a44580cb2e (127.4.158.2:45969)
I20250812 13:59:52.274261 9291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:59583
W20250812 13:59:52.518850 9322 debug-util.cc:398] Leaking SignalData structure 0x7b080007c680 after lost signal to thread 9258
W20250812 13:59:52.587031 9616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:52.587682 9616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:52.588424 9616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:52.644191 9616 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 13:59:52.644757 9616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:52.646018 9616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 13:59:52.681102 9616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:52.682425 9616 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:52.683995 9616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:52.696573 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:53.277837 9612 heartbeater.cc:499] Master 127.4.158.62:38801 was elected leader, sending a full tablet report...
W20250812 13:59:52.698859 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:52.700165 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:53.805697 9624 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1108 milliseconds
I20250812 13:59:53.805823 9616 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:53.809783 9616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:53.812343 9616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:53.813782 9616 hybrid_clock.cc:648] HybridClock initialized: now 1755007193813716 us; error 75 us; skew 500 ppm
I20250812 13:59:53.814600 9616 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:53.821105 9616 webserver.cc:480] Webserver started at http://127.4.158.3:44243/ using document root <none> and password file <none>
I20250812 13:59:53.822052 9616 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:53.822259 9616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:53.822714 9616 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:59:53.826967 9616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "58f8cfb5742a46b3b3bdcf18e8011da5"
format_stamp: "Formatted at 2025-08-12 13:59:53 on dist-test-slave-f6np"
I20250812 13:59:53.828028 9616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "58f8cfb5742a46b3b3bdcf18e8011da5"
format_stamp: "Formatted at 2025-08-12 13:59:53 on dist-test-slave-f6np"
I20250812 13:59:53.835266 9616 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250812 13:59:53.841058 9632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:53.842110 9616 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250812 13:59:53.842414 9616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "58f8cfb5742a46b3b3bdcf18e8011da5"
format_stamp: "Formatted at 2025-08-12 13:59:53 on dist-test-slave-f6np"
I20250812 13:59:53.842732 9616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:53.907191 9616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:53.908769 9616 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:53.909175 9616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:53.911840 9616 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:53.916391 9616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:59:53.916603 9616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:53.916831 9616 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:59:53.917027 9616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:54.071492 9616 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:43717
I20250812 13:59:54.071593 9744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:43717 every 8 connection(s)
I20250812 13:59:54.074182 9616 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250812 13:59:54.076149 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 9616
I20250812 13:59:54.076668 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250812 13:59:54.096549 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.4:0
--local_ip_for_outbound_sockets=127.4.158.4
--webserver_interface=127.4.158.4
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--builtin_ntp_servers=127.4.158.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250812 13:59:54.107015 9745 heartbeater.cc:344] Connected to a master server at 127.4.158.62:38801
I20250812 13:59:54.107440 9745 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:54.108429 9745 heartbeater.cc:507] Master 127.4.158.62:38801 requested a full tablet report, sending...
I20250812 13:59:54.110546 9291 ts_manager.cc:194] Registered new tserver with Master: 58f8cfb5742a46b3b3bdcf18e8011da5 (127.4.158.3:43717)
I20250812 13:59:54.111757 9291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:56183
W20250812 13:59:54.331954 9475 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6be0 after lost signal to thread 9351
W20250812 13:59:54.332763 9475 debug-util.cc:398] Leaking SignalData structure 0x7b08000acc00 after lost signal to thread 9478
W20250812 13:59:54.412765 9749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:54.413340 9749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:54.413902 9749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:54.445386 9749 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 13:59:54.445775 9749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:54.446573 9749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.4
I20250812 13:59:54.481156 9749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.4.158.4
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.4
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:54.482501 9749 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:54.484117 9749 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:54.497038 9755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:55.114730 9745 heartbeater.cc:499] Master 127.4.158.62:38801 was elected leader, sending a full tablet report...
W20250812 13:59:54.498735 9756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:55.847922 9758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:55.850780 9757 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1348 milliseconds
I20250812 13:59:55.850944 9749 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:55.852104 9749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:55.854668 9749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:55.856062 9749 hybrid_clock.cc:648] HybridClock initialized: now 1755007195856001 us; error 65 us; skew 500 ppm
I20250812 13:59:55.856820 9749 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:55.863375 9749 webserver.cc:480] Webserver started at http://127.4.158.4:34307/ using document root <none> and password file <none>
I20250812 13:59:55.864367 9749 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:55.864568 9749 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:55.865024 9749 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:59:55.869328 9749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "dd1d97b5868747658a72b25921a56add"
format_stamp: "Formatted at 2025-08-12 13:59:55 on dist-test-slave-f6np"
I20250812 13:59:55.870422 9749 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "dd1d97b5868747658a72b25921a56add"
format_stamp: "Formatted at 2025-08-12 13:59:55 on dist-test-slave-f6np"
I20250812 13:59:55.877810 9749 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20250812 13:59:55.883677 9765 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:55.884781 9749 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250812 13:59:55.885119 9749 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dd1d97b5868747658a72b25921a56add"
format_stamp: "Formatted at 2025-08-12 13:59:55 on dist-test-slave-f6np"
I20250812 13:59:55.885444 9749 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:55.948786 9749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:55.950243 9749 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:55.950657 9749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:55.953472 9749 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:55.957876 9749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:59:55.958068 9749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250812 13:59:55.958360 9749 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:59:55.958518 9749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:56.113603 9749 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.4:45205
I20250812 13:59:56.113723 9877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.4:45205 every 8 connection(s)
I20250812 13:59:56.116222 9749 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250812 13:59:56.126962 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 9749
I20250812 13:59:56.127437 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250812 13:59:56.134142 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.5:0
--local_ip_for_outbound_sockets=127.4.158.5
--webserver_interface=127.4.158.5
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--builtin_ntp_servers=127.4.158.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250812 13:59:56.139945 9878 heartbeater.cc:344] Connected to a master server at 127.4.158.62:38801
I20250812 13:59:56.140434 9878 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:56.141703 9878 heartbeater.cc:507] Master 127.4.158.62:38801 requested a full tablet report, sending...
I20250812 13:59:56.144061 9291 ts_manager.cc:194] Registered new tserver with Master: dd1d97b5868747658a72b25921a56add (127.4.158.4:45205)
I20250812 13:59:56.145265 9291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.4:46919
W20250812 13:59:56.465497 9882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 13:59:56.466001 9882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 13:59:56.466462 9882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 13:59:56.497083 9882 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250812 13:59:56.497449 9882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 13:59:56.498272 9882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.5
I20250812 13:59:56.533133 9882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.4.158.5
--webserver_port=0
--tserver_master_addrs=127.4.158.62:38801
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.5
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 13:59:56.534399 9882 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 13:59:56.535912 9882 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 13:59:56.547341 9888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 13:59:57.148437 9878 heartbeater.cc:499] Master 127.4.158.62:38801 was elected leader, sending a full tablet report...
W20250812 13:59:56.547925 9889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:57.664299 9891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 13:59:57.666920 9890 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250812 13:59:57.667093 9882 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.120s user 0.580s sys 0.539s
W20250812 13:59:57.667382 9882 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.121s user 0.580s sys 0.539s
I20250812 13:59:57.667604 9882 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 13:59:57.668593 9882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 13:59:57.671027 9882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 13:59:57.672435 9882 hybrid_clock.cc:648] HybridClock initialized: now 1755007197672380 us; error 69 us; skew 500 ppm
I20250812 13:59:57.673198 9882 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 13:59:57.680428 9882 webserver.cc:480] Webserver started at http://127.4.158.5:34631/ using document root <none> and password file <none>
I20250812 13:59:57.681412 9882 fs_manager.cc:362] Metadata directory not provided
I20250812 13:59:57.681612 9882 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 13:59:57.682080 9882 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 13:59:57.686537 9882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9f375364b2ac474c8f7d0e334d214db5"
format_stamp: "Formatted at 2025-08-12 13:59:57 on dist-test-slave-f6np"
I20250812 13:59:57.687578 9882 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9f375364b2ac474c8f7d0e334d214db5"
format_stamp: "Formatted at 2025-08-12 13:59:57 on dist-test-slave-f6np"
I20250812 13:59:57.695103 9882 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250812 13:59:57.701069 9900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:57.702124 9882 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250812 13:59:57.702419 9882 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9f375364b2ac474c8f7d0e334d214db5"
format_stamp: "Formatted at 2025-08-12 13:59:57 on dist-test-slave-f6np"
I20250812 13:59:57.702725 9882 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 13:59:57.769323 9882 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 13:59:57.770761 9882 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 13:59:57.771201 9882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 13:59:57.773577 9882 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 13:59:57.777503 9882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 13:59:57.777697 9882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:57.777940 9882 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 13:59:57.778090 9882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 13:59:57.904091 9882 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.5:42513
I20250812 13:59:57.904202 10012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.5:42513 every 8 connection(s)
I20250812 13:59:57.906594 9882 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250812 13:59:57.914336 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 9882
I20250812 13:59:57.914898 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250812 13:59:57.927577 10013 heartbeater.cc:344] Connected to a master server at 127.4.158.62:38801
I20250812 13:59:57.927981 10013 heartbeater.cc:461] Registering TS with master...
I20250812 13:59:57.928930 10013 heartbeater.cc:507] Master 127.4.158.62:38801 requested a full tablet report, sending...
I20250812 13:59:57.930960 9291 ts_manager.cc:194] Registered new tserver with Master: 9f375364b2ac474c8f7d0e334d214db5 (127.4.158.5:42513)
I20250812 13:59:57.932255 9291 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.5:35093
I20250812 13:59:57.936995 4728 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250812 13:59:57.971781 9291 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36662:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250812 13:59:58.050038 9547 tablet_service.cc:1468] Processing CreateTablet for tablet 9500edeb83dc46af8e5ec2d66be53f81 (DEFAULT_TABLE table=TestTable [id=38c52ce91bb047ab95edb60c359a8202]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:59:58.050040 9680 tablet_service.cc:1468] Processing CreateTablet for tablet 9500edeb83dc46af8e5ec2d66be53f81 (DEFAULT_TABLE table=TestTable [id=38c52ce91bb047ab95edb60c359a8202]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:59:58.051872 9680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9500edeb83dc46af8e5ec2d66be53f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:59:58.051883 9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9500edeb83dc46af8e5ec2d66be53f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:59:58.057607 9813 tablet_service.cc:1468] Processing CreateTablet for tablet 9500edeb83dc46af8e5ec2d66be53f81 (DEFAULT_TABLE table=TestTable [id=38c52ce91bb047ab95edb60c359a8202]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 13:59:58.059190 9813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9500edeb83dc46af8e5ec2d66be53f81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 13:59:58.083532 10032 tablet_bootstrap.cc:492] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5: Bootstrap starting.
I20250812 13:59:58.083528 10033 tablet_bootstrap.cc:492] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e: Bootstrap starting.
I20250812 13:59:58.091478 10033 tablet_bootstrap.cc:654] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e: Neither blocks nor log segments found. Creating new log.
I20250812 13:59:58.092536 10032 tablet_bootstrap.cc:654] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5: Neither blocks nor log segments found. Creating new log.
I20250812 13:59:58.093982 10033 log.cc:826] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:58.094771 10032 log.cc:826] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:58.096500 10034 tablet_bootstrap.cc:492] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add: Bootstrap starting.
I20250812 13:59:58.103677 10034 tablet_bootstrap.cc:654] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add: Neither blocks nor log segments found. Creating new log.
I20250812 13:59:58.105077 10032 tablet_bootstrap.cc:492] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5: No bootstrap required, opened a new log
I20250812 13:59:58.105131 10033 tablet_bootstrap.cc:492] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e: No bootstrap required, opened a new log
I20250812 13:59:58.105479 10032 ts_tablet_manager.cc:1397] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.007s
I20250812 13:59:58.105619 10033 ts_tablet_manager.cc:1397] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.010s
I20250812 13:59:58.105954 10034 log.cc:826] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add: Log is configured to *not* fsync() on all Append() calls
I20250812 13:59:58.111336 10034 tablet_bootstrap.cc:492] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add: No bootstrap required, opened a new log
I20250812 13:59:58.111894 10034 ts_tablet_manager.cc:1397] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20250812 13:59:58.130924 10033 raft_consensus.cc:357] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.131155 10032 raft_consensus.cc:357] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.132050 10032 raft_consensus.cc:383] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:59:58.132050 10033 raft_consensus.cc:383] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:59:58.132413 10033 raft_consensus.cc:738] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4191670a029c46dcab2f80a44580cb2e, State: Initialized, Role: FOLLOWER
I20250812 13:59:58.132397 10032 raft_consensus.cc:738] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f8cfb5742a46b3b3bdcf18e8011da5, State: Initialized, Role: FOLLOWER
I20250812 13:59:58.133394 10032 consensus_queue.cc:260] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.133460 10033 consensus_queue.cc:260] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.140539 10033 ts_tablet_manager.cc:1428] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e: Time spent starting tablet: real 0.035s user 0.029s sys 0.002s
I20250812 13:59:58.145148 10032 ts_tablet_manager.cc:1428] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5: Time spent starting tablet: real 0.039s user 0.027s sys 0.012s
I20250812 13:59:58.148397 10034 raft_consensus.cc:357] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.149313 10034 raft_consensus.cc:383] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 13:59:58.149627 10034 raft_consensus.cc:738] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1d97b5868747658a72b25921a56add, State: Initialized, Role: FOLLOWER
I20250812 13:59:58.150550 10034 consensus_queue.cc:260] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.160326 10034 ts_tablet_manager.cc:1428] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add: Time spent starting tablet: real 0.047s user 0.034s sys 0.000s
I20250812 13:59:58.177668 10038 raft_consensus.cc:491] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:59:58.178143 10038 raft_consensus.cc:513] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.180477 10038 leader_election.cc:290] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4191670a029c46dcab2f80a44580cb2e (127.4.158.2:45969), dd1d97b5868747658a72b25921a56add (127.4.158.4:45205)
I20250812 13:59:58.183669 10039 raft_consensus.cc:491] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 13:59:58.184242 10039 raft_consensus.cc:513] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.190241 10039 leader_election.cc:290] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f8cfb5742a46b3b3bdcf18e8011da5 (127.4.158.3:43717), dd1d97b5868747658a72b25921a56add (127.4.158.4:45205)
I20250812 13:59:58.196784 9833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9500edeb83dc46af8e5ec2d66be53f81" candidate_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1d97b5868747658a72b25921a56add" is_pre_election: true
I20250812 13:59:58.197679 9833 raft_consensus.cc:2466] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f8cfb5742a46b3b3bdcf18e8011da5 in term 0.
I20250812 13:59:58.198189 9567 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9500edeb83dc46af8e5ec2d66be53f81" candidate_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4191670a029c46dcab2f80a44580cb2e" is_pre_election: true
I20250812 13:59:58.198961 9567 raft_consensus.cc:2466] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f8cfb5742a46b3b3bdcf18e8011da5 in term 0.
I20250812 13:59:58.199457 9633 leader_election.cc:304] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f8cfb5742a46b3b3bdcf18e8011da5, dd1d97b5868747658a72b25921a56add; no voters:
I20250812 13:59:58.200420 10038 raft_consensus.cc:2802] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 13:59:58.200747 10038 raft_consensus.cc:491] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:59:58.201030 10038 raft_consensus.cc:3058] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:59:58.205302 9833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9500edeb83dc46af8e5ec2d66be53f81" candidate_uuid: "4191670a029c46dcab2f80a44580cb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1d97b5868747658a72b25921a56add" is_pre_election: true
I20250812 13:59:58.205798 9833 raft_consensus.cc:2466] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4191670a029c46dcab2f80a44580cb2e in term 0.
I20250812 13:59:58.206837 9500 leader_election.cc:304] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4191670a029c46dcab2f80a44580cb2e, dd1d97b5868747658a72b25921a56add; no voters:
I20250812 13:59:58.207187 10038 raft_consensus.cc:513] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.207295 9700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9500edeb83dc46af8e5ec2d66be53f81" candidate_uuid: "4191670a029c46dcab2f80a44580cb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" is_pre_election: true
I20250812 13:59:58.207655 10039 raft_consensus.cc:2802] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 13:59:58.207973 10039 raft_consensus.cc:491] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 13:59:58.208137 9700 raft_consensus.cc:2391] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4191670a029c46dcab2f80a44580cb2e in current term 1: Already voted for candidate 58f8cfb5742a46b3b3bdcf18e8011da5 in this term.
I20250812 13:59:58.208262 10039 raft_consensus.cc:3058] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:59:58.208951 10038 leader_election.cc:290] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [CANDIDATE]: Term 1 election: Requested vote from peers 4191670a029c46dcab2f80a44580cb2e (127.4.158.2:45969), dd1d97b5868747658a72b25921a56add (127.4.158.4:45205)
I20250812 13:59:58.209583 9567 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9500edeb83dc46af8e5ec2d66be53f81" candidate_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4191670a029c46dcab2f80a44580cb2e"
I20250812 13:59:58.209892 9833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9500edeb83dc46af8e5ec2d66be53f81" candidate_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1d97b5868747658a72b25921a56add"
I20250812 13:59:58.210381 9833 raft_consensus.cc:3058] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [term 0 FOLLOWER]: Advancing to term 1
I20250812 13:59:58.215534 10039 raft_consensus.cc:513] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.216607 9567 raft_consensus.cc:2391] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 58f8cfb5742a46b3b3bdcf18e8011da5 in current term 1: Already voted for candidate 4191670a029c46dcab2f80a44580cb2e in this term.
I20250812 13:59:58.216701 9833 raft_consensus.cc:2466] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f8cfb5742a46b3b3bdcf18e8011da5 in term 1.
I20250812 13:59:58.217900 10039 leader_election.cc:290] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [CANDIDATE]: Term 1 election: Requested vote from peers 58f8cfb5742a46b3b3bdcf18e8011da5 (127.4.158.3:43717), dd1d97b5868747658a72b25921a56add (127.4.158.4:45205)
I20250812 13:59:58.218034 9633 leader_election.cc:304] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f8cfb5742a46b3b3bdcf18e8011da5, dd1d97b5868747658a72b25921a56add; no voters:
I20250812 13:59:58.218750 10038 raft_consensus.cc:2802] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 13:59:58.218930 9700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9500edeb83dc46af8e5ec2d66be53f81" candidate_uuid: "4191670a029c46dcab2f80a44580cb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5"
I20250812 13:59:58.220806 9833 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9500edeb83dc46af8e5ec2d66be53f81" candidate_uuid: "4191670a029c46dcab2f80a44580cb2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd1d97b5868747658a72b25921a56add"
I20250812 13:59:58.221591 9833 raft_consensus.cc:2391] T 9500edeb83dc46af8e5ec2d66be53f81 P dd1d97b5868747658a72b25921a56add [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4191670a029c46dcab2f80a44580cb2e in current term 1: Already voted for candidate 58f8cfb5742a46b3b3bdcf18e8011da5 in this term.
I20250812 13:59:58.222301 10038 raft_consensus.cc:695] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [term 1 LEADER]: Becoming Leader. State: Replica: 58f8cfb5742a46b3b3bdcf18e8011da5, State: Running, Role: LEADER
I20250812 13:59:58.223104 10038 consensus_queue.cc:237] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } }
I20250812 13:59:58.233399 9501 leader_election.cc:304] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4191670a029c46dcab2f80a44580cb2e; no voters: 58f8cfb5742a46b3b3bdcf18e8011da5, dd1d97b5868747658a72b25921a56add
I20250812 13:59:58.234200 10039 raft_consensus.cc:2747] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250812 13:59:58.235925 9290 catalog_manager.cc:5582] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f8cfb5742a46b3b3bdcf18e8011da5 (127.4.158.3). New cstate: current_term: 1 leader_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f8cfb5742a46b3b3bdcf18e8011da5" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 43717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd1d97b5868747658a72b25921a56add" member_type: VOTER last_known_addr { host: "127.4.158.4" port: 45205 } health_report { overall_health: UNKNOWN } } }
W20250812 13:59:58.258097 9613 tablet.cc:2378] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:59:58.266176 4728 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250812 13:59:58.270666 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4191670a029c46dcab2f80a44580cb2e to finish bootstrapping
I20250812 13:59:58.285678 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 58f8cfb5742a46b3b3bdcf18e8011da5 to finish bootstrapping
I20250812 13:59:58.296973 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver dd1d97b5868747658a72b25921a56add to finish bootstrapping
I20250812 13:59:58.307706 4728 test_util.cc:276] Using random seed: 1956783463
I20250812 13:59:58.331141 4728 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250812 13:59:58.332024 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 9749
W20250812 13:59:58.363581 9633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.158.4:45205: connect: Connection refused (error 111)
I20250812 13:59:58.365074 9567 raft_consensus.cc:1273] T 9500edeb83dc46af8e5ec2d66be53f81 P 4191670a029c46dcab2f80a44580cb2e [term 1 FOLLOWER]: Refusing update from remote peer 58f8cfb5742a46b3b3bdcf18e8011da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 13:59:58.366696 10038 consensus_queue.cc:1035] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4191670a029c46dcab2f80a44580cb2e" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250812 13:59:58.367853 9633 consensus_peers.cc:489] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 -> Peer dd1d97b5868747658a72b25921a56add (127.4.158.4:45205): Couldn't send request to peer dd1d97b5868747658a72b25921a56add. Status: Network error: Client connection negotiation failed: client connection to 127.4.158.4:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250812 13:59:58.368815 9746 tablet.cc:2378] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 13:59:58.383198 10056 mvcc.cc:204] Tried to move back new op lower bound from 7188509484483469312 to 7188509483928375296. Current Snapshot: MvccSnapshot[applied={T|T < 7188509484483469312}]
I20250812 13:59:58.935427 10013 heartbeater.cc:499] Master 127.4.158.62:38801 was elected leader, sending a full tablet report...
W20250812 14:00:00.449802 9633 consensus_peers.cc:489] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 -> Peer dd1d97b5868747658a72b25921a56add (127.4.158.4:45205): Couldn't send request to peer dd1d97b5868747658a72b25921a56add. Status: Network error: Client connection negotiation failed: client connection to 127.4.158.4:45205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250812 14:00:00.864301 9414 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 14:00:00.866024 9547 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 14:00:00.882236 9948 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 14:00:00.884204 9680 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 14:00:02.644297 9414 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 14:00:02.649948 9680 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 14:00:02.674346 9547 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 14:00:02.676393 9948 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250812 14:00:03.019870 9633 consensus_peers.cc:489] T 9500edeb83dc46af8e5ec2d66be53f81 P 58f8cfb5742a46b3b3bdcf18e8011da5 -> Peer dd1d97b5868747658a72b25921a56add (127.4.158.4:45205): Couldn't send request to peer dd1d97b5868747658a72b25921a56add. Status: Network error: Client connection negotiation failed: client connection to 127.4.158.4:45205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250812 14:00:03.563357 9633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.158.4:45205: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250812 14:00:05.042043 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 9350
I20250812 14:00:05.062615 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 9483
I20250812 14:00:05.089390 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 9616
I20250812 14:00:05.120352 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 9882
I20250812 14:00:05.142849 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 9257
2025-08-12T14:00:05Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18650 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18650 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250812 14:00:05.196101 4728 test_util.cc:276] Using random seed: 1963671852
I20250812 14:00:05.199985 4728 ts_itest-base.cc:115] Starting cluster with:
I20250812 14:00:05.200145 4728 ts_itest-base.cc:116] --------------
I20250812 14:00:05.200258 4728 ts_itest-base.cc:117] 1 tablet servers
I20250812 14:00:05.200363 4728 ts_itest-base.cc:118] 1 replicas per TS
I20250812 14:00:05.200472 4728 ts_itest-base.cc:119] --------------
2025-08-12T14:00:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T14:00:05Z Disabled control of system clock
I20250812 14:00:05.240255 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:33109
--webserver_interface=127.4.158.62
--webserver_port=0
--builtin_ntp_servers=127.4.158.20:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:33109 with env {}
W20250812 14:00:05.535441 10154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:05.536005 10154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:05.536413 10154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:05.567348 10154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 14:00:05.567646 10154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:05.567853 10154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 14:00:05.568046 10154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 14:00:05.603458 10154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:33109
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:05.604696 10154 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:05.606253 10154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:05.616762 10160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:05.618261 10161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:06.801090 10162 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1181 milliseconds
W20250812 14:00:06.802769 10163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:06.805557 10154 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.188s user 0.000s sys 0.006s
W20250812 14:00:06.805866 10154 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.188s user 0.000s sys 0.006s
I20250812 14:00:06.806090 10154 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 14:00:06.807166 10154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:06.809741 10154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:06.811111 10154 hybrid_clock.cc:648] HybridClock initialized: now 1755007206811079 us; error 39 us; skew 500 ppm
I20250812 14:00:06.811901 10154 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:06.819308 10154 webserver.cc:480] Webserver started at http://127.4.158.62:43921/ using document root <none> and password file <none>
I20250812 14:00:06.820211 10154 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:06.820436 10154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:06.820870 10154 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 14:00:06.825290 10154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7766c2718ced4bc2b4892d8cece24e8c"
format_stamp: "Formatted at 2025-08-12 14:00:06 on dist-test-slave-f6np"
I20250812 14:00:06.826447 10154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7766c2718ced4bc2b4892d8cece24e8c"
format_stamp: "Formatted at 2025-08-12 14:00:06 on dist-test-slave-f6np"
I20250812 14:00:06.834239 10154 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20250812 14:00:06.840380 10170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:06.841450 10154 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250812 14:00:06.841764 10154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7766c2718ced4bc2b4892d8cece24e8c"
format_stamp: "Formatted at 2025-08-12 14:00:06 on dist-test-slave-f6np"
I20250812 14:00:06.842124 10154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:06.900568 10154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:06.902026 10154 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:06.902457 10154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:06.969460 10154 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:33109
I20250812 14:00:06.969550 10221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:33109 every 8 connection(s)
I20250812 14:00:06.972146 10154 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250812 14:00:06.977005 10222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:06.979002 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 10154
I20250812 14:00:06.979493 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250812 14:00:06.997496 10222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c: Bootstrap starting.
I20250812 14:00:07.003497 10222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:07.005100 10222 log.cc:826] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:07.009553 10222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c: No bootstrap required, opened a new log
I20250812 14:00:07.026690 10222 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7766c2718ced4bc2b4892d8cece24e8c" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33109 } }
I20250812 14:00:07.027499 10222 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:07.027763 10222 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7766c2718ced4bc2b4892d8cece24e8c, State: Initialized, Role: FOLLOWER
I20250812 14:00:07.028512 10222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7766c2718ced4bc2b4892d8cece24e8c" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33109 } }
I20250812 14:00:07.028990 10222 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 14:00:07.029361 10222 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 14:00:07.029709 10222 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:07.033890 10222 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7766c2718ced4bc2b4892d8cece24e8c" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33109 } }
I20250812 14:00:07.034634 10222 leader_election.cc:304] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7766c2718ced4bc2b4892d8cece24e8c; no voters:
I20250812 14:00:07.036334 10222 leader_election.cc:290] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 14:00:07.037040 10227 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 1 FOLLOWER]: Leader election won for term 1
I20250812 14:00:07.039095 10227 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [term 1 LEADER]: Becoming Leader. State: Replica: 7766c2718ced4bc2b4892d8cece24e8c, State: Running, Role: LEADER
I20250812 14:00:07.039855 10227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7766c2718ced4bc2b4892d8cece24e8c" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33109 } }
I20250812 14:00:07.041026 10222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [sys.catalog]: configured and running, proceeding with master startup.
I20250812 14:00:07.047098 10229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7766c2718ced4bc2b4892d8cece24e8c. Latest consensus state: current_term: 1 leader_uuid: "7766c2718ced4bc2b4892d8cece24e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7766c2718ced4bc2b4892d8cece24e8c" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33109 } } }
I20250812 14:00:07.047933 10229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [sys.catalog]: This master's current role is: LEADER
I20250812 14:00:07.047242 10228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7766c2718ced4bc2b4892d8cece24e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7766c2718ced4bc2b4892d8cece24e8c" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33109 } } }
I20250812 14:00:07.049175 10228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c [sys.catalog]: This master's current role is: LEADER
I20250812 14:00:07.056386 10235 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 14:00:07.069168 10235 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 14:00:07.084976 10235 catalog_manager.cc:1349] Generated new cluster ID: 3434c27b1c2e4859a47bebea772879e7
I20250812 14:00:07.085358 10235 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 14:00:07.107411 10235 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 14:00:07.108886 10235 catalog_manager.cc:1506] Loading token signing keys...
I20250812 14:00:07.131894 10235 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7766c2718ced4bc2b4892d8cece24e8c: Generated new TSK 0
I20250812 14:00:07.132774 10235 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 14:00:07.157608 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:0
--local_ip_for_outbound_sockets=127.4.158.1
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:33109
--builtin_ntp_servers=127.4.158.20:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250812 14:00:07.453871 10247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:07.454373 10247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:07.454876 10247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:07.486896 10247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:07.487747 10247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 14:00:07.523890 10247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:33109
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:07.525193 10247 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:07.526782 10247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:07.538763 10253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:08.941094 10252 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 10247
W20250812 14:00:09.133126 10252 kernel_stack_watchdog.cc:198] Thread 10247 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 14:00:07.539927 10254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:09.134514 10247 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.594s user 0.559s sys 0.992s
W20250812 14:00:09.134918 10247 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.595s user 0.559s sys 0.993s
W20250812 14:00:09.136072 10256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:09.138417 10255 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1594 milliseconds
I20250812 14:00:09.138458 10247 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 14:00:09.139849 10247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:09.142398 10247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:09.143918 10247 hybrid_clock.cc:648] HybridClock initialized: now 1755007209143854 us; error 67 us; skew 500 ppm
I20250812 14:00:09.144927 10247 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:09.151893 10247 webserver.cc:480] Webserver started at http://127.4.158.1:39893/ using document root <none> and password file <none>
I20250812 14:00:09.153084 10247 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:09.153343 10247 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:09.153900 10247 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 14:00:09.160176 10247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8638b6f23a2f45ada7065fb6c8ea4b58"
format_stamp: "Formatted at 2025-08-12 14:00:09 on dist-test-slave-f6np"
I20250812 14:00:09.161599 10247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8638b6f23a2f45ada7065fb6c8ea4b58"
format_stamp: "Formatted at 2025-08-12 14:00:09 on dist-test-slave-f6np"
I20250812 14:00:09.170320 10247 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250812 14:00:09.177291 10263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:09.178455 10247 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250812 14:00:09.178840 10247 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8638b6f23a2f45ada7065fb6c8ea4b58"
format_stamp: "Formatted at 2025-08-12 14:00:09 on dist-test-slave-f6np"
I20250812 14:00:09.179252 10247 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:09.279325 10247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:09.280855 10247 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:09.281328 10247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:09.284014 10247 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 14:00:09.288424 10247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 14:00:09.288632 10247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:09.289016 10247 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 14:00:09.289197 10247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:09.439785 10247 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:42931
I20250812 14:00:09.439950 10375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:42931 every 8 connection(s)
I20250812 14:00:09.442415 10247 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250812 14:00:09.448375 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 10247
I20250812 14:00:09.448899 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755007066725366-4728-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250812 14:00:09.467214 10376 heartbeater.cc:344] Connected to a master server at 127.4.158.62:33109
I20250812 14:00:09.467716 10376 heartbeater.cc:461] Registering TS with master...
I20250812 14:00:09.468986 10376 heartbeater.cc:507] Master 127.4.158.62:33109 requested a full tablet report, sending...
I20250812 14:00:09.471758 10187 ts_manager.cc:194] Registered new tserver with Master: 8638b6f23a2f45ada7065fb6c8ea4b58 (127.4.158.1:42931)
I20250812 14:00:09.474315 10187 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:52691
I20250812 14:00:09.483098 4728 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250812 14:00:09.516664 10187 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49310:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250812 14:00:09.571604 10311 tablet_service.cc:1468] Processing CreateTablet for tablet 073ba51ee17b4a2d88530878d3a7ccb9 (DEFAULT_TABLE table=TestTable [id=a9a87e65dffc46cdbffaa08a1bf97fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 14:00:09.573051 10311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 073ba51ee17b4a2d88530878d3a7ccb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:09.592612 10391 tablet_bootstrap.cc:492] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58: Bootstrap starting.
I20250812 14:00:09.597970 10391 tablet_bootstrap.cc:654] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:09.599701 10391 log.cc:826] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:09.604153 10391 tablet_bootstrap.cc:492] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58: No bootstrap required, opened a new log
I20250812 14:00:09.604542 10391 ts_tablet_manager.cc:1397] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250812 14:00:09.623111 10391 raft_consensus.cc:357] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8638b6f23a2f45ada7065fb6c8ea4b58" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 42931 } }
I20250812 14:00:09.623685 10391 raft_consensus.cc:383] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:09.623953 10391 raft_consensus.cc:738] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8638b6f23a2f45ada7065fb6c8ea4b58, State: Initialized, Role: FOLLOWER
I20250812 14:00:09.624663 10391 consensus_queue.cc:260] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8638b6f23a2f45ada7065fb6c8ea4b58" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 42931 } }
I20250812 14:00:09.625182 10391 raft_consensus.cc:397] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 14:00:09.625442 10391 raft_consensus.cc:491] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 14:00:09.625756 10391 raft_consensus.cc:3058] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:09.630321 10391 raft_consensus.cc:513] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8638b6f23a2f45ada7065fb6c8ea4b58" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 42931 } }
I20250812 14:00:09.630986 10391 leader_election.cc:304] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8638b6f23a2f45ada7065fb6c8ea4b58; no voters:
I20250812 14:00:09.632620 10391 leader_election.cc:290] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 14:00:09.632970 10393 raft_consensus.cc:2802] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 14:00:09.635303 10393 raft_consensus.cc:695] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [term 1 LEADER]: Becoming Leader. State: Replica: 8638b6f23a2f45ada7065fb6c8ea4b58, State: Running, Role: LEADER
I20250812 14:00:09.635867 10376 heartbeater.cc:499] Master 127.4.158.62:33109 was elected leader, sending a full tablet report...
I20250812 14:00:09.636194 10393 consensus_queue.cc:237] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8638b6f23a2f45ada7065fb6c8ea4b58" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 42931 } }
I20250812 14:00:09.636611 10391 ts_tablet_manager.cc:1428] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58: Time spent starting tablet: real 0.032s user 0.017s sys 0.015s
I20250812 14:00:09.648231 10187 catalog_manager.cc:5582] T 073ba51ee17b4a2d88530878d3a7ccb9 P 8638b6f23a2f45ada7065fb6c8ea4b58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8638b6f23a2f45ada7065fb6c8ea4b58 (127.4.158.1). New cstate: current_term: 1 leader_uuid: "8638b6f23a2f45ada7065fb6c8ea4b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8638b6f23a2f45ada7065fb6c8ea4b58" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 42931 } health_report { overall_health: HEALTHY } } }
I20250812 14:00:09.673763 4728 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250812 14:00:09.676952 4728 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8638b6f23a2f45ada7065fb6c8ea4b58 to finish bootstrapping
I20250812 14:00:12.231683 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 10247
I20250812 14:00:12.251885 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 10154
2025-08-12T14:00:12Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7107 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7107 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250812 14:00:12.304431 4728 test_util.cc:276] Using random seed: 1970780180
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-12T14:00:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-12T14:00:12Z Disabled control of system clock
I20250812 14:00:12.354957 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:33535
--webserver_interface=127.4.158.62
--webserver_port=0
--builtin_ntp_servers=127.4.158.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:33535 with env {}
W20250812 14:00:12.651687 10419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:12.652313 10419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:12.652765 10419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:12.684197 10419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 14:00:12.684511 10419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:12.684759 10419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 14:00:12.684990 10419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 14:00:12.720551 10419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:33535
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:33535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:12.721879 10419 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:12.723464 10419 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:12.734313 10425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:12.734938 10426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:13.908380 10428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:13.910920 10427 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1170 milliseconds
W20250812 14:00:13.910938 10419 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.177s user 0.452s sys 0.721s
W20250812 14:00:13.911317 10419 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.177s user 0.453s sys 0.721s
I20250812 14:00:13.911559 10419 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 14:00:13.912622 10419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:13.915222 10419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:13.916568 10419 hybrid_clock.cc:648] HybridClock initialized: now 1755007213916540 us; error 39 us; skew 500 ppm
I20250812 14:00:13.917348 10419 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:13.924552 10419 webserver.cc:480] Webserver started at http://127.4.158.62:46499/ using document root <none> and password file <none>
I20250812 14:00:13.925488 10419 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:13.925702 10419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:13.926159 10419 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 14:00:13.930532 10419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/instance:
uuid: "9159ebf0a30e4f449c86f39365c11430"
format_stamp: "Formatted at 2025-08-12 14:00:13 on dist-test-slave-f6np"
I20250812 14:00:13.931614 10419 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal/instance:
uuid: "9159ebf0a30e4f449c86f39365c11430"
format_stamp: "Formatted at 2025-08-12 14:00:13 on dist-test-slave-f6np"
I20250812 14:00:13.939498 10419 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250812 14:00:13.945492 10435 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:13.946729 10419 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250812 14:00:13.947052 10419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
uuid: "9159ebf0a30e4f449c86f39365c11430"
format_stamp: "Formatted at 2025-08-12 14:00:13 on dist-test-slave-f6np"
I20250812 14:00:13.947402 10419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:14.011341 10419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:14.012813 10419 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:14.013283 10419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:14.080816 10419 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:33535
I20250812 14:00:14.080888 10486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:33535 every 8 connection(s)
I20250812 14:00:14.083477 10419 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/info.pb
I20250812 14:00:14.088347 10487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:14.088294 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 10419
I20250812 14:00:14.088696 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal/instance
I20250812 14:00:14.113591 10487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430: Bootstrap starting.
I20250812 14:00:14.118979 10487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:14.120783 10487 log.cc:826] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:14.125366 10487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430: No bootstrap required, opened a new log
I20250812 14:00:14.143280 10487 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9159ebf0a30e4f449c86f39365c11430" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } }
I20250812 14:00:14.143937 10487 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:14.144150 10487 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9159ebf0a30e4f449c86f39365c11430, State: Initialized, Role: FOLLOWER
I20250812 14:00:14.144752 10487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9159ebf0a30e4f449c86f39365c11430" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } }
I20250812 14:00:14.145226 10487 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 14:00:14.145447 10487 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 14:00:14.145700 10487 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:14.150367 10487 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9159ebf0a30e4f449c86f39365c11430" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } }
I20250812 14:00:14.151023 10487 leader_election.cc:304] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9159ebf0a30e4f449c86f39365c11430; no voters:
I20250812 14:00:14.152588 10487 leader_election.cc:290] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 14:00:14.153322 10492 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 14:00:14.155258 10492 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [term 1 LEADER]: Becoming Leader. State: Replica: 9159ebf0a30e4f449c86f39365c11430, State: Running, Role: LEADER
I20250812 14:00:14.156019 10492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9159ebf0a30e4f449c86f39365c11430" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } }
I20250812 14:00:14.156980 10487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 14:00:14.163154 10494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9159ebf0a30e4f449c86f39365c11430. Latest consensus state: current_term: 1 leader_uuid: "9159ebf0a30e4f449c86f39365c11430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9159ebf0a30e4f449c86f39365c11430" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } } }
I20250812 14:00:14.163266 10493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9159ebf0a30e4f449c86f39365c11430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9159ebf0a30e4f449c86f39365c11430" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } } }
I20250812 14:00:14.163849 10494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [sys.catalog]: This master's current role is: LEADER
I20250812 14:00:14.163926 10493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430 [sys.catalog]: This master's current role is: LEADER
I20250812 14:00:14.168162 10500 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 14:00:14.179808 10500 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 14:00:14.195439 10500 catalog_manager.cc:1349] Generated new cluster ID: 25015b10be404443a1aafa9a5f0e8650
I20250812 14:00:14.195739 10500 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 14:00:14.209474 10500 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 14:00:14.210911 10500 catalog_manager.cc:1506] Loading token signing keys...
I20250812 14:00:14.236111 10500 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9159ebf0a30e4f449c86f39365c11430: Generated new TSK 0
I20250812 14:00:14.237030 10500 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250812 14:00:14.261380 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:0
--local_ip_for_outbound_sockets=127.4.158.1
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:33535
--builtin_ntp_servers=127.4.158.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250812 14:00:14.565243 10511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:14.565742 10511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:14.566260 10511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:14.599010 10511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:14.599872 10511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 14:00:14.636672 10511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=0
--tserver_master_addrs=127.4.158.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:14.638005 10511 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:14.639617 10511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:14.651810 10517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:14.657140 10520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:14.654747 10518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:16.115159 10519 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1458 milliseconds
I20250812 14:00:16.115338 10511 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 14:00:16.116520 10511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:16.118734 10511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:16.120044 10511 hybrid_clock.cc:648] HybridClock initialized: now 1755007216119999 us; error 64 us; skew 500 ppm
I20250812 14:00:16.120826 10511 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:16.127136 10511 webserver.cc:480] Webserver started at http://127.4.158.1:46357/ using document root <none> and password file <none>
I20250812 14:00:16.128060 10511 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:16.128253 10511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:16.128732 10511 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 14:00:16.133091 10511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/instance:
uuid: "aec70b8ae82f49f18b52c8e3e81f8750"
format_stamp: "Formatted at 2025-08-12 14:00:16 on dist-test-slave-f6np"
I20250812 14:00:16.134213 10511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal/instance:
uuid: "aec70b8ae82f49f18b52c8e3e81f8750"
format_stamp: "Formatted at 2025-08-12 14:00:16 on dist-test-slave-f6np"
I20250812 14:00:16.141019 10511 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250812 14:00:16.146495 10527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:16.147490 10511 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250812 14:00:16.147791 10511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
uuid: "aec70b8ae82f49f18b52c8e3e81f8750"
format_stamp: "Formatted at 2025-08-12 14:00:16 on dist-test-slave-f6np"
I20250812 14:00:16.148116 10511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:16.195331 10511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:16.196760 10511 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:16.197189 10511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:16.199664 10511 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 14:00:16.203824 10511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 14:00:16.204032 10511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:16.204324 10511 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 14:00:16.204474 10511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:16.331393 10511 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:35683
I20250812 14:00:16.331504 10639 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:35683 every 8 connection(s)
I20250812 14:00:16.334050 10511 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/info.pb
I20250812 14:00:16.343552 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 10511
I20250812 14:00:16.343971 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal/instance
I20250812 14:00:16.350307 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:0
--local_ip_for_outbound_sockets=127.4.158.2
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:33535
--builtin_ntp_servers=127.4.158.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 14:00:16.364368 10640 heartbeater.cc:344] Connected to a master server at 127.4.158.62:33535
I20250812 14:00:16.364817 10640 heartbeater.cc:461] Registering TS with master...
I20250812 14:00:16.365904 10640 heartbeater.cc:507] Master 127.4.158.62:33535 requested a full tablet report, sending...
I20250812 14:00:16.368551 10452 ts_manager.cc:194] Registered new tserver with Master: aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1:35683)
I20250812 14:00:16.370677 10452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:49505
W20250812 14:00:16.646147 10644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:16.646653 10644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:16.647154 10644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:16.678758 10644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:16.679612 10644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 14:00:16.715024 10644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=0
--tserver_master_addrs=127.4.158.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:16.716362 10644 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:16.718031 10644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:16.730077 10650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 14:00:17.374308 10640 heartbeater.cc:499] Master 127.4.158.62:33535 was elected leader, sending a full tablet report...
W20250812 14:00:16.731084 10651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:16.734833 10653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:17.838945 10652 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
I20250812 14:00:17.839087 10644 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 14:00:17.840234 10644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:17.842664 10644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:17.844067 10644 hybrid_clock.cc:648] HybridClock initialized: now 1755007217844031 us; error 57 us; skew 500 ppm
I20250812 14:00:17.844846 10644 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:17.850651 10644 webserver.cc:480] Webserver started at http://127.4.158.2:39339/ using document root <none> and password file <none>
I20250812 14:00:17.851590 10644 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:17.851794 10644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:17.852272 10644 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 14:00:17.856607 10644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/instance:
uuid: "13daf01be9154f5a8303269075497a0b"
format_stamp: "Formatted at 2025-08-12 14:00:17 on dist-test-slave-f6np"
I20250812 14:00:17.857693 10644 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal/instance:
uuid: "13daf01be9154f5a8303269075497a0b"
format_stamp: "Formatted at 2025-08-12 14:00:17 on dist-test-slave-f6np"
I20250812 14:00:17.864555 10644 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250812 14:00:17.869962 10661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:17.871014 10644 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250812 14:00:17.871323 10644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
uuid: "13daf01be9154f5a8303269075497a0b"
format_stamp: "Formatted at 2025-08-12 14:00:17 on dist-test-slave-f6np"
I20250812 14:00:17.871661 10644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:17.916945 10644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:17.918421 10644 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:17.918848 10644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:17.921277 10644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 14:00:17.925213 10644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 14:00:17.925426 10644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:17.925647 10644 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 14:00:17.925796 10644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:18.050926 10644 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:43103
I20250812 14:00:18.051038 10773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:43103 every 8 connection(s)
I20250812 14:00:18.053483 10644 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/info.pb
I20250812 14:00:18.057829 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 10644
I20250812 14:00:18.058562 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal/instance
I20250812 14:00:18.064891 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:0
--local_ip_for_outbound_sockets=127.4.158.3
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:33535
--builtin_ntp_servers=127.4.158.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 14:00:18.075055 10774 heartbeater.cc:344] Connected to a master server at 127.4.158.62:33535
I20250812 14:00:18.075563 10774 heartbeater.cc:461] Registering TS with master...
I20250812 14:00:18.076814 10774 heartbeater.cc:507] Master 127.4.158.62:33535 requested a full tablet report, sending...
I20250812 14:00:18.079222 10452 ts_manager.cc:194] Registered new tserver with Master: 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103)
I20250812 14:00:18.080854 10452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:36889
W20250812 14:00:18.367762 10778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:18.368252 10778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:18.368782 10778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:18.403575 10778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:18.404732 10778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 14:00:18.439713 10778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=0
--tserver_master_addrs=127.4.158.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:18.440980 10778 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:18.442582 10778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:18.454325 10784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 14:00:19.084688 10774 heartbeater.cc:499] Master 127.4.158.62:33535 was elected leader, sending a full tablet report...
W20250812 14:00:19.859560 10783 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 10778
W20250812 14:00:19.864135 10783 kernel_stack_watchdog.cc:198] Thread 10778 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250812 14:00:19.864383 10778 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.410s user 0.575s sys 0.825s
W20250812 14:00:18.454795 10785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:19.864735 10778 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.410s user 0.575s sys 0.825s
W20250812 14:00:19.866695 10787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:19.869695 10786 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1411 milliseconds
I20250812 14:00:19.869769 10778 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 14:00:19.870970 10778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:19.872992 10778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:19.874327 10778 hybrid_clock.cc:648] HybridClock initialized: now 1755007219874296 us; error 33 us; skew 500 ppm
I20250812 14:00:19.875097 10778 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:19.880838 10778 webserver.cc:480] Webserver started at http://127.4.158.3:39039/ using document root <none> and password file <none>
I20250812 14:00:19.881752 10778 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:19.882007 10778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:19.882433 10778 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 14:00:19.886747 10778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/instance:
uuid: "2f184ca7f701451691bdd85c13b4019b"
format_stamp: "Formatted at 2025-08-12 14:00:19 on dist-test-slave-f6np"
I20250812 14:00:19.887804 10778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal/instance:
uuid: "2f184ca7f701451691bdd85c13b4019b"
format_stamp: "Formatted at 2025-08-12 14:00:19 on dist-test-slave-f6np"
I20250812 14:00:19.894878 10778 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20250812 14:00:19.900242 10794 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:19.901216 10778 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250812 14:00:19.901526 10778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
uuid: "2f184ca7f701451691bdd85c13b4019b"
format_stamp: "Formatted at 2025-08-12 14:00:19 on dist-test-slave-f6np"
I20250812 14:00:19.901839 10778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:19.958858 10778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:19.960273 10778 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:19.960702 10778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:19.963205 10778 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 14:00:19.967141 10778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250812 14:00:19.967355 10778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:19.967586 10778 ts_tablet_manager.cc:610] Registered 0 tablets
I20250812 14:00:19.967743 10778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:20.101395 10778 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:44787
I20250812 14:00:20.101511 10906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:44787 every 8 connection(s)
I20250812 14:00:20.103901 10778 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/info.pb
I20250812 14:00:20.111017 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 10778
I20250812 14:00:20.111533 4728 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal/instance
I20250812 14:00:20.125038 10907 heartbeater.cc:344] Connected to a master server at 127.4.158.62:33535
I20250812 14:00:20.125468 10907 heartbeater.cc:461] Registering TS with master...
I20250812 14:00:20.126636 10907 heartbeater.cc:507] Master 127.4.158.62:33535 requested a full tablet report, sending...
I20250812 14:00:20.128862 10452 ts_manager.cc:194] Registered new tserver with Master: 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787)
I20250812 14:00:20.130246 10452 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:46297
I20250812 14:00:20.132050 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 14:00:20.161917 4728 test_util.cc:276] Using random seed: 1978637667
I20250812 14:00:20.200874 10451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36386:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250812 14:00:20.203377 10451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 14:00:20.247498 10842 tablet_service.cc:1468] Processing CreateTablet for tablet 4328770079f244428d897a06b135abf7 (DEFAULT_TABLE table=pre_rebuild [id=716f1575478945b085c228e50477293d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 14:00:20.249536 10842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4328770079f244428d897a06b135abf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:20.249017 10709 tablet_service.cc:1468] Processing CreateTablet for tablet 4328770079f244428d897a06b135abf7 (DEFAULT_TABLE table=pre_rebuild [id=716f1575478945b085c228e50477293d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 14:00:20.250779 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4328770079f244428d897a06b135abf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:20.258729 10575 tablet_service.cc:1468] Processing CreateTablet for tablet 4328770079f244428d897a06b135abf7 (DEFAULT_TABLE table=pre_rebuild [id=716f1575478945b085c228e50477293d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 14:00:20.260661 10575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4328770079f244428d897a06b135abf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:20.272403 10931 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Bootstrap starting.
I20250812 14:00:20.275880 10932 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Bootstrap starting.
I20250812 14:00:20.281209 10931 tablet_bootstrap.cc:654] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:20.283004 10932 tablet_bootstrap.cc:654] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:20.283452 10931 log.cc:826] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:20.284726 10933 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Bootstrap starting.
I20250812 14:00:20.285341 10932 log.cc:826] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:20.290305 10931 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: No bootstrap required, opened a new log
I20250812 14:00:20.290364 10932 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: No bootstrap required, opened a new log
I20250812 14:00:20.290747 10932 ts_tablet_manager.cc:1397] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20250812 14:00:20.290784 10931 ts_tablet_manager.cc:1397] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20250812 14:00:20.292446 10933 tablet_bootstrap.cc:654] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:20.294816 10933 log.cc:826] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:20.300196 10933 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: No bootstrap required, opened a new log
I20250812 14:00:20.300750 10933 ts_tablet_manager.cc:1397] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20250812 14:00:20.310451 10931 raft_consensus.cc:357] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.311137 10931 raft_consensus.cc:383] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:20.311540 10931 raft_consensus.cc:738] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f184ca7f701451691bdd85c13b4019b, State: Initialized, Role: FOLLOWER
I20250812 14:00:20.312474 10931 consensus_queue.cc:260] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.313752 10932 raft_consensus.cc:357] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.314719 10932 raft_consensus.cc:383] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:20.315151 10932 raft_consensus.cc:738] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13daf01be9154f5a8303269075497a0b, State: Initialized, Role: FOLLOWER
I20250812 14:00:20.316596 10907 heartbeater.cc:499] Master 127.4.158.62:33535 was elected leader, sending a full tablet report...
I20250812 14:00:20.316179 10932 consensus_queue.cc:260] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.317986 10931 ts_tablet_manager.cc:1428] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Time spent starting tablet: real 0.027s user 0.022s sys 0.005s
I20250812 14:00:20.320750 10932 ts_tablet_manager.cc:1428] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Time spent starting tablet: real 0.030s user 0.023s sys 0.005s
I20250812 14:00:20.325079 10933 raft_consensus.cc:357] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.325896 10933 raft_consensus.cc:383] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:20.326231 10933 raft_consensus.cc:738] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec70b8ae82f49f18b52c8e3e81f8750, State: Initialized, Role: FOLLOWER
I20250812 14:00:20.327090 10933 consensus_queue.cc:260] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.331130 10933 ts_tablet_manager.cc:1428] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Time spent starting tablet: real 0.030s user 0.027s sys 0.002s
W20250812 14:00:20.342378 10641 tablet.cc:2378] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 14:00:20.358193 10908 tablet.cc:2378] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 14:00:20.508857 10939 raft_consensus.cc:491] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 14:00:20.509377 10939 raft_consensus.cc:513] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.511847 10939 leader_election.cc:290] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787), 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103)
I20250812 14:00:20.517194 10937 raft_consensus.cc:491] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 14:00:20.517778 10937 raft_consensus.cc:513] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.522516 10937 leader_election.cc:290] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103), aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1:35683)
I20250812 14:00:20.523836 10862 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f184ca7f701451691bdd85c13b4019b" is_pre_election: true
I20250812 14:00:20.524679 10862 raft_consensus.cc:2466] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec70b8ae82f49f18b52c8e3e81f8750 in term 0.
I20250812 14:00:20.529357 10729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13daf01be9154f5a8303269075497a0b" is_pre_election: true
I20250812 14:00:20.529922 10528 leader_election.cc:304] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f184ca7f701451691bdd85c13b4019b, aec70b8ae82f49f18b52c8e3e81f8750; no voters:
I20250812 14:00:20.530267 10729 raft_consensus.cc:2466] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec70b8ae82f49f18b52c8e3e81f8750 in term 0.
I20250812 14:00:20.530757 10939 raft_consensus.cc:2802] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 14:00:20.531090 10939 raft_consensus.cc:491] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 14:00:20.531426 10939 raft_consensus.cc:3058] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:20.534216 10729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "2f184ca7f701451691bdd85c13b4019b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13daf01be9154f5a8303269075497a0b" is_pre_election: true
I20250812 14:00:20.534758 10729 raft_consensus.cc:2466] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f184ca7f701451691bdd85c13b4019b in term 0.
I20250812 14:00:20.535844 10795 leader_election.cc:304] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13daf01be9154f5a8303269075497a0b, 2f184ca7f701451691bdd85c13b4019b; no voters:
I20250812 14:00:20.536726 10937 raft_consensus.cc:2802] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 14:00:20.537050 10937 raft_consensus.cc:491] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 14:00:20.537303 10937 raft_consensus.cc:3058] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:20.537307 10939 raft_consensus.cc:513] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.538791 10939 leader_election.cc:290] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 1 election: Requested vote from peers 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787), 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103)
I20250812 14:00:20.539296 10595 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "2f184ca7f701451691bdd85c13b4019b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" is_pre_election: true
I20250812 14:00:20.539980 10862 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f184ca7f701451691bdd85c13b4019b"
I20250812 14:00:20.540014 10729 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13daf01be9154f5a8303269075497a0b"
I20250812 14:00:20.540268 10595 raft_consensus.cc:2391] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f184ca7f701451691bdd85c13b4019b in current term 1: Already voted for candidate aec70b8ae82f49f18b52c8e3e81f8750 in this term.
I20250812 14:00:20.540493 10729 raft_consensus.cc:3058] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:20.542903 10937 raft_consensus.cc:513] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:20.543926 10862 raft_consensus.cc:2391] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aec70b8ae82f49f18b52c8e3e81f8750 in current term 1: Already voted for candidate 2f184ca7f701451691bdd85c13b4019b in this term.
I20250812 14:00:20.544440 10937 leader_election.cc:290] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [CANDIDATE]: Term 1 election: Requested vote from peers 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103), aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1:35683)
I20250812 14:00:20.545516 10595 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "2f184ca7f701451691bdd85c13b4019b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec70b8ae82f49f18b52c8e3e81f8750"
I20250812 14:00:20.545900 10728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "2f184ca7f701451691bdd85c13b4019b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13daf01be9154f5a8303269075497a0b"
I20250812 14:00:20.546217 10595 raft_consensus.cc:2391] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f184ca7f701451691bdd85c13b4019b in current term 1: Already voted for candidate aec70b8ae82f49f18b52c8e3e81f8750 in this term.
I20250812 14:00:20.546118 10729 raft_consensus.cc:2466] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec70b8ae82f49f18b52c8e3e81f8750 in term 1.
I20250812 14:00:20.547443 10528 leader_election.cc:304] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13daf01be9154f5a8303269075497a0b, aec70b8ae82f49f18b52c8e3e81f8750; no voters:
I20250812 14:00:20.548038 10939 raft_consensus.cc:2802] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 14:00:20.548410 10795 leader_election.cc:304] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f184ca7f701451691bdd85c13b4019b; no voters: 13daf01be9154f5a8303269075497a0b, aec70b8ae82f49f18b52c8e3e81f8750
I20250812 14:00:20.548983 10937 raft_consensus.cc:2747] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250812 14:00:20.552246 10939 raft_consensus.cc:695] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 LEADER]: Becoming Leader. State: Replica: aec70b8ae82f49f18b52c8e3e81f8750, State: Running, Role: LEADER
I20250812 14:00:20.553149 10939 consensus_queue.cc:237] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
W20250812 14:00:20.560174 10775 tablet.cc:2378] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 14:00:20.563918 10449 catalog_manager.cc:5582] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 reported cstate change: term changed from 0 to 1, leader changed from <none> to aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1). New cstate: current_term: 1 leader_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } health_report { overall_health: HEALTHY } } }
I20250812 14:00:20.694680 10728 raft_consensus.cc:1273] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 1 FOLLOWER]: Refusing update from remote peer aec70b8ae82f49f18b52c8e3e81f8750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 14:00:20.695063 10862 raft_consensus.cc:1273] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Refusing update from remote peer aec70b8ae82f49f18b52c8e3e81f8750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 14:00:20.696274 10944 consensus_queue.cc:1035] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250812 14:00:20.696966 10939 consensus_queue.cc:1035] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 14:00:20.726145 10952 mvcc.cc:204] Tried to move back new op lower bound from 7188509575948541952 to 7188509575392468992. Current Snapshot: MvccSnapshot[applied={T|T < 7188509575948541952}]
I20250812 14:00:25.764460 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 10419
W20250812 14:00:25.795553 10797 connection.cc:537] client connection to 127.4.158.62:33535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250812 14:00:25.796540 10907 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:33535 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250812 14:00:25.797367 10530 connection.cc:537] client connection to 127.4.158.62:33535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250812 14:00:25.798245 10640 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:33535 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250812 14:00:25.807088 10774 heartbeater.cc:646] Failed to heartbeat to 127.4.158.62:33535 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.158.62:33535: connect: Connection refused (error 111)
W20250812 14:00:26.175733 10985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:26.176364 10985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:26.209439 10985 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250812 14:00:27.639284 10985 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.384s user 0.573s sys 0.795s
W20250812 14:00:27.639668 10985 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.384s user 0.573s sys 0.795s
W20250812 14:00:27.716786 10903 debug-util.cc:398] Leaking SignalData structure 0x7b08000be100 after lost signal to thread 10779
W20250812 14:00:27.717794 10903 debug-util.cc:398] Leaking SignalData structure 0x7b08000cbbe0 after lost signal to thread 10906
I20250812 14:00:27.783622 10985 minidump.cc:252] Setting minidump size limit to 20M
I20250812 14:00:27.786396 10985 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:27.787925 10985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:27.802536 11016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:27.804764 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:27.941183 11022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 14:00:27.942049 10985 server_base.cc:1047] running on GCE node
I20250812 14:00:27.943082 10985 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250812 14:00:27.943571 10985 hybrid_clock.cc:648] HybridClock initialized: now 1755007227943548 us; error 340516 us; skew 500 ppm
I20250812 14:00:27.944267 10985 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:27.949265 10985 webserver.cc:480] Webserver started at http://0.0.0.0:44099/ using document root <none> and password file <none>
I20250812 14:00:27.950169 10985 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:27.950384 10985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:27.950798 10985 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250812 14:00:27.955022 10985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/instance:
uuid: "38b98ed424c04929acc43a674d86e1e5"
format_stamp: "Formatted at 2025-08-12 14:00:27 on dist-test-slave-f6np"
I20250812 14:00:27.956101 10985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal/instance:
uuid: "38b98ed424c04929acc43a674d86e1e5"
format_stamp: "Formatted at 2025-08-12 14:00:27 on dist-test-slave-f6np"
I20250812 14:00:27.962134 10985 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250812 14:00:27.967016 11027 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:27.967950 10985 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250812 14:00:27.968256 10985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
uuid: "38b98ed424c04929acc43a674d86e1e5"
format_stamp: "Formatted at 2025-08-12 14:00:27 on dist-test-slave-f6np"
I20250812 14:00:27.968566 10985 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:28.264324 10985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:28.265753 10985 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:28.266242 10985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:28.271615 10985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:28.286473 10985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Bootstrap starting.
I20250812 14:00:28.291333 10985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:28.293004 10985 log.cc:826] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:28.297319 10985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: No bootstrap required, opened a new log
I20250812 14:00:28.313273 10985 raft_consensus.cc:357] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER }
I20250812 14:00:28.313755 10985 raft_consensus.cc:383] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:28.314030 10985 raft_consensus.cc:738] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b98ed424c04929acc43a674d86e1e5, State: Initialized, Role: FOLLOWER
I20250812 14:00:28.314694 10985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER }
I20250812 14:00:28.315160 10985 raft_consensus.cc:397] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 14:00:28.315407 10985 raft_consensus.cc:491] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 14:00:28.315691 10985 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:28.319598 10985 raft_consensus.cc:513] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER }
I20250812 14:00:28.320254 10985 leader_election.cc:304] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b98ed424c04929acc43a674d86e1e5; no voters:
I20250812 14:00:28.322275 10985 leader_election.cc:290] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250812 14:00:28.322528 11034 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 FOLLOWER]: Leader election won for term 1
I20250812 14:00:28.326036 11034 raft_consensus.cc:695] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 38b98ed424c04929acc43a674d86e1e5, State: Running, Role: LEADER
I20250812 14:00:28.326859 11034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER }
I20250812 14:00:28.332530 11035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38b98ed424c04929acc43a674d86e1e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER } }
I20250812 14:00:28.333091 11035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: This master's current role is: LEADER
I20250812 14:00:28.333488 11036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b98ed424c04929acc43a674d86e1e5. Latest consensus state: current_term: 1 leader_uuid: "38b98ed424c04929acc43a674d86e1e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER } }
I20250812 14:00:28.334002 11036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: This master's current role is: LEADER
I20250812 14:00:28.350346 10985 tablet_replica.cc:331] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: stopping tablet replica
I20250812 14:00:28.350958 10985 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 LEADER]: Raft consensus shutting down.
I20250812 14:00:28.351441 10985 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250812 14:00:28.353957 10985 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250812 14:00:28.354566 10985 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250812 14:00:28.383666 10985 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250812 14:00:29.412288 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 10511
I20250812 14:00:29.449334 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 10644
I20250812 14:00:29.476358 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 10778
I20250812 14:00:29.507332 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:33535
--webserver_interface=127.4.158.62
--webserver_port=46499
--builtin_ntp_servers=127.4.158.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.158.62:33535 with env {}
W20250812 14:00:29.805692 11044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:29.806334 11044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:29.806789 11044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:29.838276 11044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250812 14:00:29.838617 11044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:29.838882 11044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250812 14:00:29.839116 11044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250812 14:00:29.874814 11044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.158.62:33535
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.158.62:33535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.158.62
--webserver_port=46499
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:29.876139 11044 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:29.877766 11044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:29.888188 11050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:29.888589 11051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:30.993949 11053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:30.997267 11052 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1107 milliseconds
W20250812 14:00:30.997872 11044 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.110s user 0.400s sys 0.708s
W20250812 14:00:30.998174 11044 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.110s user 0.402s sys 0.708s
I20250812 14:00:30.998386 11044 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 14:00:30.999461 11044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:31.004516 11044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:31.005904 11044 hybrid_clock.cc:648] HybridClock initialized: now 1755007231005865 us; error 54 us; skew 500 ppm
I20250812 14:00:31.006716 11044 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:31.014102 11044 webserver.cc:480] Webserver started at http://127.4.158.62:46499/ using document root <none> and password file <none>
I20250812 14:00:31.015277 11044 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:31.015589 11044 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:31.024621 11044 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250812 14:00:31.029567 11060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:31.030973 11044 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250812 14:00:31.031306 11044 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
uuid: "38b98ed424c04929acc43a674d86e1e5"
format_stamp: "Formatted at 2025-08-12 14:00:27 on dist-test-slave-f6np"
I20250812 14:00:31.033280 11044 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:31.119036 11044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:31.120575 11044 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:31.121042 11044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:31.190286 11044 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.62:33535
I20250812 14:00:31.190331 11111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.62:33535 every 8 connection(s)
I20250812 14:00:31.193135 11044 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/info.pb
I20250812 14:00:31.198518 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 11044
I20250812 14:00:31.200009 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.1:35683
--local_ip_for_outbound_sockets=127.4.158.1
--tserver_master_addrs=127.4.158.62:33535
--webserver_port=46357
--webserver_interface=127.4.158.1
--builtin_ntp_servers=127.4.158.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 14:00:31.203066 11112 sys_catalog.cc:263] Verifying existing consensus state
I20250812 14:00:31.217712 11112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Bootstrap starting.
I20250812 14:00:31.231417 11112 log.cc:826] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:31.246261 11112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 14:00:31.247248 11112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Bootstrap complete.
I20250812 14:00:31.277535 11112 raft_consensus.cc:357] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } }
I20250812 14:00:31.278481 11112 raft_consensus.cc:738] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b98ed424c04929acc43a674d86e1e5, State: Initialized, Role: FOLLOWER
I20250812 14:00:31.279420 11112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } }
I20250812 14:00:31.280128 11112 raft_consensus.cc:397] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250812 14:00:31.280452 11112 raft_consensus.cc:491] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250812 14:00:31.280812 11112 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 1 FOLLOWER]: Advancing to term 2
I20250812 14:00:31.286840 11112 raft_consensus.cc:513] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } }
I20250812 14:00:31.287705 11112 leader_election.cc:304] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b98ed424c04929acc43a674d86e1e5; no voters:
I20250812 14:00:31.289702 11112 leader_election.cc:290] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250812 14:00:31.290182 11116 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 14:00:31.293020 11116 raft_consensus.cc:695] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [term 2 LEADER]: Becoming Leader. State: Replica: 38b98ed424c04929acc43a674d86e1e5, State: Running, Role: LEADER
I20250812 14:00:31.293764 11116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } }
I20250812 14:00:31.294437 11112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: configured and running, proceeding with master startup.
I20250812 14:00:31.300585 11117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38b98ed424c04929acc43a674d86e1e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } } }
I20250812 14:00:31.300501 11118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b98ed424c04929acc43a674d86e1e5. Latest consensus state: current_term: 2 leader_uuid: "38b98ed424c04929acc43a674d86e1e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b98ed424c04929acc43a674d86e1e5" member_type: VOTER last_known_addr { host: "127.4.158.62" port: 33535 } } }
I20250812 14:00:31.301210 11117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: This master's current role is: LEADER
I20250812 14:00:31.301555 11118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5 [sys.catalog]: This master's current role is: LEADER
I20250812 14:00:31.309024 11122 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250812 14:00:31.321718 11122 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=49323b7ca9534b66be5528bb5190d08d]
I20250812 14:00:31.328665 11122 tablet_loader.cc:96] loaded metadata for tablet 4328770079f244428d897a06b135abf7 (table pre_rebuild [id=49323b7ca9534b66be5528bb5190d08d])
I20250812 14:00:31.330343 11122 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250812 14:00:31.361366 11122 catalog_manager.cc:1349] Generated new cluster ID: 0bda9f84137d400d8ffe6da6a0462fcf
I20250812 14:00:31.361666 11122 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250812 14:00:31.401120 11122 catalog_manager.cc:1372] Generated new certificate authority record
I20250812 14:00:31.403208 11122 catalog_manager.cc:1506] Loading token signing keys...
I20250812 14:00:31.428772 11122 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Generated new TSK 0
I20250812 14:00:31.429898 11122 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250812 14:00:31.571532 11114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:31.572062 11114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:31.572547 11114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:31.604159 11114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:31.605031 11114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.1
I20250812 14:00:31.643303 11114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.1:35683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.158.1
--webserver_port=46357
--tserver_master_addrs=127.4.158.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.1
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:31.644703 11114 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:31.646364 11114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:31.659252 11140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:31.659910 11141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:32.921653 11114 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.264s user 0.477s sys 0.777s
W20250812 14:00:32.922099 11114 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.265s user 0.479s sys 0.777s
I20250812 14:00:32.923763 11114 server_base.cc:1047] running on GCE node
W20250812 14:00:32.925450 11145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 14:00:32.926913 11114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:32.929690 11114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:32.931206 11114 hybrid_clock.cc:648] HybridClock initialized: now 1755007232931122 us; error 82 us; skew 500 ppm
I20250812 14:00:32.932281 11114 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:32.940989 11114 webserver.cc:480] Webserver started at http://127.4.158.1:46357/ using document root <none> and password file <none>
I20250812 14:00:32.942317 11114 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:32.942617 11114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:32.953416 11114 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20250812 14:00:32.960011 11150 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:32.961369 11114 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250812 14:00:32.961766 11114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
uuid: "aec70b8ae82f49f18b52c8e3e81f8750"
format_stamp: "Formatted at 2025-08-12 14:00:16 on dist-test-slave-f6np"
I20250812 14:00:32.964525 11114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:33.054242 11114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:33.055716 11114 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:33.056139 11114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:33.059010 11114 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 14:00:33.064989 11157 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 14:00:33.072611 11114 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 14:00:33.072849 11114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.001s
I20250812 14:00:33.073135 11114 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 14:00:33.077795 11114 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 14:00:33.078080 11114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250812 14:00:33.078439 11157 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Bootstrap starting.
I20250812 14:00:33.276013 11114 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.1:35683
I20250812 14:00:33.276199 11263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.1:35683 every 8 connection(s)
I20250812 14:00:33.278782 11114 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/info.pb
I20250812 14:00:33.286710 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 11114
I20250812 14:00:33.288729 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.2:43103
--local_ip_for_outbound_sockets=127.4.158.2
--tserver_master_addrs=127.4.158.62:33535
--webserver_port=39339
--webserver_interface=127.4.158.2
--builtin_ntp_servers=127.4.158.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 14:00:33.345780 11264 heartbeater.cc:344] Connected to a master server at 127.4.158.62:33535
I20250812 14:00:33.346321 11264 heartbeater.cc:461] Registering TS with master...
I20250812 14:00:33.347537 11264 heartbeater.cc:507] Master 127.4.158.62:33535 requested a full tablet report, sending...
I20250812 14:00:33.352337 11077 ts_manager.cc:194] Registered new tserver with Master: aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1:35683)
I20250812 14:00:33.359673 11077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.1:41579
I20250812 14:00:33.399685 11157 log.cc:826] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Log is configured to *not* fsync() on all Append() calls
W20250812 14:00:33.640583 11268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:33.641083 11268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:33.641564 11268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:33.673341 11268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:33.674269 11268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.2
I20250812 14:00:33.709401 11268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.2:43103
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.158.2
--webserver_port=39339
--tserver_master_addrs=127.4.158.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.2
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:33.710760 11268 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:33.712388 11268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:33.724630 11275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 14:00:34.364507 11264 heartbeater.cc:499] Master 127.4.158.62:33535 was elected leader, sending a full tablet report...
W20250812 14:00:33.728549 11278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:33.726655 11276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250812 14:00:34.993641 11277 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1266 milliseconds
I20250812 14:00:34.993731 11268 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250812 14:00:34.994939 11268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:34.997488 11268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:34.998889 11268 hybrid_clock.cc:648] HybridClock initialized: now 1755007234998825 us; error 79 us; skew 500 ppm
I20250812 14:00:34.999661 11268 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:35.006300 11268 webserver.cc:480] Webserver started at http://127.4.158.2:39339/ using document root <none> and password file <none>
I20250812 14:00:35.007462 11268 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:35.007686 11268 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:35.015556 11268 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250812 14:00:35.020282 11285 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:35.021306 11268 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250812 14:00:35.021606 11268 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
uuid: "13daf01be9154f5a8303269075497a0b"
format_stamp: "Formatted at 2025-08-12 14:00:17 on dist-test-slave-f6np"
I20250812 14:00:35.023522 11268 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:35.072911 11268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:35.074391 11268 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:35.074798 11268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:35.077553 11268 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 14:00:35.083513 11292 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 14:00:35.090929 11268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 14:00:35.091156 11268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250812 14:00:35.091423 11268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 14:00:35.096005 11268 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 14:00:35.096252 11268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250812 14:00:35.096619 11292 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Bootstrap starting.
I20250812 14:00:35.313625 11268 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.2:43103
I20250812 14:00:35.313827 11398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.2:43103 every 8 connection(s)
I20250812 14:00:35.317305 11268 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/info.pb
I20250812 14:00:35.320250 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 11268
I20250812 14:00:35.322099 4728 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
/tmp/dist-test-tasktZztFx/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.158.3:44787
--local_ip_for_outbound_sockets=127.4.158.3
--tserver_master_addrs=127.4.158.62:33535
--webserver_port=39039
--webserver_interface=127.4.158.3
--builtin_ntp_servers=127.4.158.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250812 14:00:35.369547 11399 heartbeater.cc:344] Connected to a master server at 127.4.158.62:33535
I20250812 14:00:35.370388 11399 heartbeater.cc:461] Registering TS with master...
I20250812 14:00:35.372404 11399 heartbeater.cc:507] Master 127.4.158.62:33535 requested a full tablet report, sending...
I20250812 14:00:35.377012 11077 ts_manager.cc:194] Registered new tserver with Master: 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103)
I20250812 14:00:35.380409 11077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.2:39903
I20250812 14:00:35.395588 11292 log.cc:826] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:35.757409 11157 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 14:00:35.758237 11157 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Bootstrap complete.
I20250812 14:00:35.759387 11157 ts_tablet_manager.cc:1397] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Time spent bootstrapping tablet: real 2.681s user 2.565s sys 0.093s
I20250812 14:00:35.770102 11157 raft_consensus.cc:357] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:35.772148 11157 raft_consensus.cc:738] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec70b8ae82f49f18b52c8e3e81f8750, State: Initialized, Role: FOLLOWER
I20250812 14:00:35.772912 11157 consensus_queue.cc:260] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:35.776181 11157 ts_tablet_manager.cc:1428] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
W20250812 14:00:35.785205 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250812 14:00:35.785840 11402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250812 14:00:35.786605 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250812 14:00:35.840029 11402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250812 14:00:35.840905 11402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.158.3
I20250812 14:00:35.875710 11402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.158.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.158.3:44787
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.158.3
--webserver_port=39039
--tserver_master_addrs=127.4.158.62:33535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.158.3
--log_dir=/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 12 Aug 2025 13:43:24 UTC on 5fd53c4cbb9d
build id 7533
TSAN enabled
I20250812 14:00:35.877034 11402 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250812 14:00:35.878682 11402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250812 14:00:35.893677 11412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 14:00:36.384620 11399 heartbeater.cc:499] Master 127.4.158.62:33535 was elected leader, sending a full tablet report...
W20250812 14:00:35.895967 11414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 14:00:37.218271 11402 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250812 14:00:37.218855 11413 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1324 milliseconds
W20250812 14:00:35.895067 11411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250812 14:00:37.222878 11402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250812 14:00:37.225621 11402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250812 14:00:37.227094 11402 hybrid_clock.cc:648] HybridClock initialized: now 1755007237227077 us; error 65 us; skew 500 ppm
I20250812 14:00:37.227867 11402 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250812 14:00:37.234295 11402 webserver.cc:480] Webserver started at http://127.4.158.3:39039/ using document root <none> and password file <none>
I20250812 14:00:37.235208 11402 fs_manager.cc:362] Metadata directory not provided
I20250812 14:00:37.235424 11402 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250812 14:00:37.243232 11402 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250812 14:00:37.248147 11422 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250812 14:00:37.249191 11402 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250812 14:00:37.249492 11402 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
uuid: "2f184ca7f701451691bdd85c13b4019b"
format_stamp: "Formatted at 2025-08-12 14:00:19 on dist-test-slave-f6np"
I20250812 14:00:37.251367 11402 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250812 14:00:37.303133 11402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250812 14:00:37.304539 11402 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250812 14:00:37.304953 11402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250812 14:00:37.307789 11402 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250812 14:00:37.314648 11429 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250812 14:00:37.325757 11402 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250812 14:00:37.326098 11402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.003s
I20250812 14:00:37.326483 11402 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250812 14:00:37.331403 11402 ts_tablet_manager.cc:610] Registered 1 tablets
I20250812 14:00:37.331606 11402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250812 14:00:37.332080 11429 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Bootstrap starting.
I20250812 14:00:37.513057 11402 rpc_server.cc:307] RPC server started. Bound to: 127.4.158.3:44787
I20250812 14:00:37.513157 11535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.158.3:44787 every 8 connection(s)
I20250812 14:00:37.515919 11402 server_base.cc:1179] Dumped server information to /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/info.pb
I20250812 14:00:37.519356 4728 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu as pid 11402
I20250812 14:00:37.536835 11537 raft_consensus.cc:491] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 14:00:37.537705 11537 raft_consensus.cc:513] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:37.555130 11537 leader_election.cc:290] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787), 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103)
I20250812 14:00:37.642469 11536 heartbeater.cc:344] Connected to a master server at 127.4.158.62:33535
I20250812 14:00:37.643087 11536 heartbeater.cc:461] Registering TS with master...
I20250812 14:00:37.644541 11536 heartbeater.cc:507] Master 127.4.158.62:33535 requested a full tablet report, sending...
I20250812 14:00:37.647943 11429 log.cc:826] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Log is configured to *not* fsync() on all Append() calls
I20250812 14:00:37.649935 11077 ts_manager.cc:194] Registered new tserver with Master: 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787)
I20250812 14:00:37.653188 11077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.158.3:48041
I20250812 14:00:37.679067 4728 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250812 14:00:37.683600 11354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "13daf01be9154f5a8303269075497a0b" is_pre_election: true
W20250812 14:00:37.706486 11151 leader_election.cc:343] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103): Illegal state: must be running to vote when last-logged opid is not known
I20250812 14:00:37.682308 11491 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "2f184ca7f701451691bdd85c13b4019b" is_pre_election: true
W20250812 14:00:37.709075 11151 leader_election.cc:343] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787): Illegal state: must be running to vote when last-logged opid is not known
I20250812 14:00:37.709481 11151 leader_election.cc:304] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aec70b8ae82f49f18b52c8e3e81f8750; no voters: 13daf01be9154f5a8303269075497a0b, 2f184ca7f701451691bdd85c13b4019b
I20250812 14:00:37.710359 11537 raft_consensus.cc:2747] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250812 14:00:38.132656 11292 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 14:00:38.133498 11292 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Bootstrap complete.
I20250812 14:00:38.134667 11292 ts_tablet_manager.cc:1397] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Time spent bootstrapping tablet: real 3.038s user 2.949s sys 0.080s
I20250812 14:00:38.139806 11292 raft_consensus.cc:357] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:38.142295 11292 raft_consensus.cc:738] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13daf01be9154f5a8303269075497a0b, State: Initialized, Role: FOLLOWER
I20250812 14:00:38.142975 11292 consensus_queue.cc:260] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:38.145921 11292 ts_tablet_manager.cc:1428] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20250812 14:00:38.660575 11536 heartbeater.cc:499] Master 127.4.158.62:33535 was elected leader, sending a full tablet report...
I20250812 14:00:39.356807 11554 raft_consensus.cc:491] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 14:00:39.357208 11554 raft_consensus.cc:513] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:39.358861 11554 leader_election.cc:290] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787), 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103)
I20250812 14:00:39.359658 11491 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "2f184ca7f701451691bdd85c13b4019b" is_pre_election: true
I20250812 14:00:39.359956 11354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "13daf01be9154f5a8303269075497a0b" is_pre_election: true
I20250812 14:00:39.360618 11354 raft_consensus.cc:2466] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec70b8ae82f49f18b52c8e3e81f8750 in term 1.
W20250812 14:00:39.360787 11151 leader_election.cc:343] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787): Illegal state: must be running to vote when last-logged opid is not known
I20250812 14:00:39.361598 11151 leader_election.cc:304] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13daf01be9154f5a8303269075497a0b, aec70b8ae82f49f18b52c8e3e81f8750; no voters: 2f184ca7f701451691bdd85c13b4019b
I20250812 14:00:39.362305 11554 raft_consensus.cc:2802] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250812 14:00:39.362568 11554 raft_consensus.cc:491] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 14:00:39.362802 11554 raft_consensus.cc:3058] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Advancing to term 2
I20250812 14:00:39.368569 11554 raft_consensus.cc:513] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:39.369837 11554 leader_election.cc:290] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 election: Requested vote from peers 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787), 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103)
I20250812 14:00:39.370656 11491 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "2f184ca7f701451691bdd85c13b4019b"
I20250812 14:00:39.371008 11354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4328770079f244428d897a06b135abf7" candidate_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "13daf01be9154f5a8303269075497a0b"
I20250812 14:00:39.371510 11354 raft_consensus.cc:3058] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 1 FOLLOWER]: Advancing to term 2
W20250812 14:00:39.371704 11151 leader_election.cc:343] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787): Illegal state: must be running to vote when last-logged opid is not known
I20250812 14:00:39.377480 11354 raft_consensus.cc:2466] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec70b8ae82f49f18b52c8e3e81f8750 in term 2.
I20250812 14:00:39.378299 11151 leader_election.cc:304] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13daf01be9154f5a8303269075497a0b, aec70b8ae82f49f18b52c8e3e81f8750; no voters: 2f184ca7f701451691bdd85c13b4019b
I20250812 14:00:39.378888 11554 raft_consensus.cc:2802] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 2 FOLLOWER]: Leader election won for term 2
I20250812 14:00:39.380380 11554 raft_consensus.cc:695] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 2 LEADER]: Becoming Leader. State: Replica: aec70b8ae82f49f18b52c8e3e81f8750, State: Running, Role: LEADER
I20250812 14:00:39.381346 11554 consensus_queue.cc:237] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:39.391039 11077 catalog_manager.cc:5582] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 reported cstate change: term changed from 0 to 2, leader changed from <none> to aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1), VOTER 13daf01be9154f5a8303269075497a0b (127.4.158.2) added, VOTER 2f184ca7f701451691bdd85c13b4019b (127.4.158.3) added, VOTER aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1) added. New cstate: current_term: 2 leader_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } health_report { overall_health: HEALTHY } } }
I20250812 14:00:39.583742 11429 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250812 14:00:39.584539 11429 tablet_bootstrap.cc:492] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Bootstrap complete.
I20250812 14:00:39.585657 11429 ts_tablet_manager.cc:1397] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Time spent bootstrapping tablet: real 2.254s user 2.158s sys 0.056s
I20250812 14:00:39.591179 11429 raft_consensus.cc:357] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:39.593093 11429 raft_consensus.cc:738] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f184ca7f701451691bdd85c13b4019b, State: Initialized, Role: FOLLOWER
I20250812 14:00:39.593760 11429 consensus_queue.cc:260] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:39.596244 11429 ts_tablet_manager.cc:1428] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Time spent starting tablet: real 0.010s user 0.013s sys 0.000s
I20250812 14:00:39.779920 11354 raft_consensus.cc:1273] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 2 FOLLOWER]: Refusing update from remote peer aec70b8ae82f49f18b52c8e3e81f8750: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250812 14:00:39.781307 11554 consensus_queue.cc:1035] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250812 14:00:39.786614 11491 raft_consensus.cc:3058] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Advancing to term 2
I20250812 14:00:39.796571 11491 raft_consensus.cc:1273] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 2 FOLLOWER]: Refusing update from remote peer aec70b8ae82f49f18b52c8e3e81f8750: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250812 14:00:39.798919 11554 consensus_queue.cc:1035] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250812 14:00:39.828768 11219 consensus_queue.cc:237] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:39.832397 11354 raft_consensus.cc:1273] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 2 FOLLOWER]: Refusing update from remote peer aec70b8ae82f49f18b52c8e3e81f8750: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250812 14:00:39.833436 11554 consensus_queue.cc:1035] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250812 14:00:39.838171 11555 raft_consensus.cc:2953] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 2f184ca7f701451691bdd85c13b4019b (127.4.158.3) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } } }
I20250812 14:00:39.839433 11354 raft_consensus.cc:2953] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 2f184ca7f701451691bdd85c13b4019b (127.4.158.3) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } } }
I20250812 14:00:39.847713 11061 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4328770079f244428d897a06b135abf7 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250812 14:00:39.850140 11076 catalog_manager.cc:5582] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 reported cstate change: config changed from index -1 to 207, VOTER 2f184ca7f701451691bdd85c13b4019b (127.4.158.3) evicted. New cstate: current_term: 2 leader_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } health_report { overall_health: HEALTHY } } }
I20250812 14:00:39.862546 11219 consensus_queue.cc:237] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:39.864634 11555 raft_consensus.cc:2953] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 13daf01be9154f5a8303269075497a0b (127.4.158.2) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } } }
I20250812 14:00:39.872545 11061 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4328770079f244428d897a06b135abf7 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250812 14:00:39.875715 11077 catalog_manager.cc:5582] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 reported cstate change: config changed from index 207 to 208, VOTER 13daf01be9154f5a8303269075497a0b (127.4.158.2) evicted. New cstate: current_term: 2 leader_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } health_report { overall_health: HEALTHY } } }
I20250812 14:00:39.872881 11471 tablet_service.cc:1515] Processing DeleteTablet for tablet 4328770079f244428d897a06b135abf7 with delete_type TABLET_DATA_TOMBSTONED (TS 2f184ca7f701451691bdd85c13b4019b not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:36452
I20250812 14:00:39.879165 11575 tablet_replica.cc:331] stopping tablet replica
I20250812 14:00:39.879786 11575 raft_consensus.cc:2241] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250812 14:00:39.880331 11575 raft_consensus.cc:2270] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250812 14:00:39.896098 11334 tablet_service.cc:1515] Processing DeleteTablet for tablet 4328770079f244428d897a06b135abf7 with delete_type TABLET_DATA_TOMBSTONED (TS 13daf01be9154f5a8303269075497a0b not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:53994
I20250812 14:00:39.898918 11578 tablet_replica.cc:331] stopping tablet replica
I20250812 14:00:39.899736 11578 raft_consensus.cc:2241] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250812 14:00:39.900426 11578 raft_consensus.cc:2270] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250812 14:00:39.913352 11575 ts_tablet_manager.cc:1905] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250812 14:00:39.928970 11575 ts_tablet_manager.cc:1918] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250812 14:00:39.929510 11575 log.cc:1199] T 4328770079f244428d897a06b135abf7 P 2f184ca7f701451691bdd85c13b4019b: Deleting WAL directory at /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/wal/wals/4328770079f244428d897a06b135abf7
I20250812 14:00:39.931416 11061 catalog_manager.cc:4928] TS 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787): tablet 4328770079f244428d897a06b135abf7 (table pre_rebuild [id=49323b7ca9534b66be5528bb5190d08d]) successfully deleted
I20250812 14:00:39.932580 11578 ts_tablet_manager.cc:1905] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250812 14:00:39.947768 11578 ts_tablet_manager.cc:1918] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250812 14:00:39.948213 11578 log.cc:1199] T 4328770079f244428d897a06b135abf7 P 13daf01be9154f5a8303269075497a0b: Deleting WAL directory at /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/wal/wals/4328770079f244428d897a06b135abf7
I20250812 14:00:39.950006 11061 catalog_manager.cc:4928] TS 13daf01be9154f5a8303269075497a0b (127.4.158.2:43103): tablet 4328770079f244428d897a06b135abf7 (table pre_rebuild [id=49323b7ca9534b66be5528bb5190d08d]) successfully deleted
W20250812 14:00:40.110885 4728 scanner-internal.cc:458] Time spent opening tablet: real 2.368s user 0.007s sys 0.001s
I20250812 14:00:40.710127 11471 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 14:00:40.710409 11334 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 14:00:40.724350 11199 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
38b98ed424c04929acc43a674d86e1e5 | 127.4.158.62:33535 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.4.158.20:40709 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
13daf01be9154f5a8303269075497a0b | 127.4.158.2:43103 | HEALTHY | <none> | 0 | 0
2f184ca7f701451691bdd85c13b4019b | 127.4.158.3:44787 | HEALTHY | <none> | 0 | 0
aec70b8ae82f49f18b52c8e3e81f8750 | 127.4.158.1:35683 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.4.158.1 | experimental | 127.4.158.1:35683
local_ip_for_outbound_sockets | 127.4.158.2 | experimental | 127.4.158.2:43103
local_ip_for_outbound_sockets | 127.4.158.3 | experimental | 127.4.158.3:44787
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/info.pb | hidden | 127.4.158.1:35683
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/info.pb | hidden | 127.4.158.2:43103
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/info.pb | hidden | 127.4.158.3:44787
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.4.158.20:40709 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250812 14:00:40.965708 4728 log_verifier.cc:126] Checking tablet 4328770079f244428d897a06b135abf7
I20250812 14:00:41.233247 4728 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 4328770079f244428d897a06b135abf7
I20250812 14:00:41.235502 11076 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44560:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250812 14:00:41.236059 11076 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44560:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250812 14:00:41.248173 11076 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 38b98ed424c04929acc43a674d86e1e5: Sending DeleteTablet for 1 replicas of tablet 4328770079f244428d897a06b135abf7
I20250812 14:00:41.250125 4728 test_util.cc:276] Using random seed: 1999725881
I20250812 14:00:41.250092 11199 tablet_service.cc:1515] Processing DeleteTablet for tablet 4328770079f244428d897a06b135abf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-12 14:00:41 UTC) from {username='slave'} at 127.0.0.1:45192
I20250812 14:00:41.257900 11610 tablet_replica.cc:331] stopping tablet replica
I20250812 14:00:41.258813 11610 raft_consensus.cc:2241] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 2 LEADER]: Raft consensus shutting down.
I20250812 14:00:41.259490 11610 raft_consensus.cc:2270] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250812 14:00:41.302394 11610 ts_tablet_manager.cc:1905] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250812 14:00:41.316500 11076 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44596:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250812 14:00:41.317981 11610 ts_tablet_manager.cc:1918] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250812 14:00:41.318444 11610 log.cc:1199] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Deleting WAL directory at /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/wal/wals/4328770079f244428d897a06b135abf7
I20250812 14:00:41.319362 11610 ts_tablet_manager.cc:1939] T 4328770079f244428d897a06b135abf7 P aec70b8ae82f49f18b52c8e3e81f8750: Deleting consensus metadata
W20250812 14:00:41.319989 11076 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250812 14:00:41.322340 11061 catalog_manager.cc:4928] TS aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1:35683): tablet 4328770079f244428d897a06b135abf7 (table pre_rebuild [id=49323b7ca9534b66be5528bb5190d08d]) successfully deleted
I20250812 14:00:41.347209 11334 tablet_service.cc:1468] Processing CreateTablet for tablet 5aefe60d220349e88d56f0342f9b238e (DEFAULT_TABLE table=post_rebuild [id=c791dd5698884e0faedf8e563556309d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 14:00:41.348553 11334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aefe60d220349e88d56f0342f9b238e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:41.353179 11199 tablet_service.cc:1468] Processing CreateTablet for tablet 5aefe60d220349e88d56f0342f9b238e (DEFAULT_TABLE table=post_rebuild [id=c791dd5698884e0faedf8e563556309d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 14:00:41.353125 11471 tablet_service.cc:1468] Processing CreateTablet for tablet 5aefe60d220349e88d56f0342f9b238e (DEFAULT_TABLE table=post_rebuild [id=c791dd5698884e0faedf8e563556309d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250812 14:00:41.354439 11199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aefe60d220349e88d56f0342f9b238e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:41.354425 11471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aefe60d220349e88d56f0342f9b238e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250812 14:00:41.386547 11617 tablet_bootstrap.cc:492] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b: Bootstrap starting.
I20250812 14:00:41.388314 11619 tablet_bootstrap.cc:492] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750: Bootstrap starting.
I20250812 14:00:41.395327 11618 tablet_bootstrap.cc:492] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b: Bootstrap starting.
I20250812 14:00:41.395828 11619 tablet_bootstrap.cc:654] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:41.396731 11617 tablet_bootstrap.cc:654] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:41.401297 11618 tablet_bootstrap.cc:654] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b: Neither blocks nor log segments found. Creating new log.
I20250812 14:00:41.408453 11617 tablet_bootstrap.cc:492] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b: No bootstrap required, opened a new log
I20250812 14:00:41.409004 11617 ts_tablet_manager.cc:1397] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.005s
I20250812 14:00:41.412415 11618 tablet_bootstrap.cc:492] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b: No bootstrap required, opened a new log
I20250812 14:00:41.412915 11618 ts_tablet_manager.cc:1397] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250812 14:00:41.415557 11618 raft_consensus.cc:357] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.415524 11617 raft_consensus.cc:357] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.416231 11617 raft_consensus.cc:383] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:41.416325 11618 raft_consensus.cc:383] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:41.416570 11617 raft_consensus.cc:738] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f184ca7f701451691bdd85c13b4019b, State: Initialized, Role: FOLLOWER
I20250812 14:00:41.416713 11618 raft_consensus.cc:738] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13daf01be9154f5a8303269075497a0b, State: Initialized, Role: FOLLOWER
I20250812 14:00:41.417493 11619 tablet_bootstrap.cc:492] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750: No bootstrap required, opened a new log
I20250812 14:00:41.417362 11617 consensus_queue.cc:260] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.417987 11619 ts_tablet_manager.cc:1397] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.015s
I20250812 14:00:41.417518 11618 consensus_queue.cc:260] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.420796 11619 raft_consensus.cc:357] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.421530 11619 raft_consensus.cc:383] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250812 14:00:41.421928 11619 raft_consensus.cc:738] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec70b8ae82f49f18b52c8e3e81f8750, State: Initialized, Role: FOLLOWER
I20250812 14:00:41.422797 11619 consensus_queue.cc:260] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.426774 11618 ts_tablet_manager.cc:1428] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20250812 14:00:41.431496 11617 ts_tablet_manager.cc:1428] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b: Time spent starting tablet: real 0.022s user 0.006s sys 0.011s
I20250812 14:00:41.437830 11619 ts_tablet_manager.cc:1428] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750: Time spent starting tablet: real 0.020s user 0.009s sys 0.003s
I20250812 14:00:41.470353 11624 raft_consensus.cc:491] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250812 14:00:41.470860 11624 raft_consensus.cc:513] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.473987 11624 leader_election.cc:290] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787), aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1:35683)
I20250812 14:00:41.487386 11491 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aefe60d220349e88d56f0342f9b238e" candidate_uuid: "13daf01be9154f5a8303269075497a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f184ca7f701451691bdd85c13b4019b" is_pre_election: true
I20250812 14:00:41.488237 11491 raft_consensus.cc:2466] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13daf01be9154f5a8303269075497a0b in term 0.
I20250812 14:00:41.489677 11286 leader_election.cc:304] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13daf01be9154f5a8303269075497a0b, 2f184ca7f701451691bdd85c13b4019b; no voters:
I20250812 14:00:41.490617 11624 raft_consensus.cc:2802] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250812 14:00:41.490955 11624 raft_consensus.cc:491] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250812 14:00:41.490962 11219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aefe60d220349e88d56f0342f9b238e" candidate_uuid: "13daf01be9154f5a8303269075497a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" is_pre_election: true
I20250812 14:00:41.491256 11624 raft_consensus.cc:3058] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:41.491786 11219 raft_consensus.cc:2466] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13daf01be9154f5a8303269075497a0b in term 0.
I20250812 14:00:41.498015 11624 raft_consensus.cc:513] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.499822 11624 leader_election.cc:290] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [CANDIDATE]: Term 1 election: Requested vote from peers 2f184ca7f701451691bdd85c13b4019b (127.4.158.3:44787), aec70b8ae82f49f18b52c8e3e81f8750 (127.4.158.1:35683)
I20250812 14:00:41.500708 11491 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aefe60d220349e88d56f0342f9b238e" candidate_uuid: "13daf01be9154f5a8303269075497a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f184ca7f701451691bdd85c13b4019b"
I20250812 14:00:41.500933 11219 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aefe60d220349e88d56f0342f9b238e" candidate_uuid: "13daf01be9154f5a8303269075497a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec70b8ae82f49f18b52c8e3e81f8750"
I20250812 14:00:41.501250 11491 raft_consensus.cc:3058] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:41.501402 11219 raft_consensus.cc:3058] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750 [term 0 FOLLOWER]: Advancing to term 1
I20250812 14:00:41.505879 11219 raft_consensus.cc:2466] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13daf01be9154f5a8303269075497a0b in term 1.
I20250812 14:00:41.506781 11286 leader_election.cc:304] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13daf01be9154f5a8303269075497a0b, aec70b8ae82f49f18b52c8e3e81f8750; no voters:
I20250812 14:00:41.507391 11624 raft_consensus.cc:2802] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 1 FOLLOWER]: Leader election won for term 1
I20250812 14:00:41.507658 11491 raft_consensus.cc:2466] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13daf01be9154f5a8303269075497a0b in term 1.
I20250812 14:00:41.509373 11624 raft_consensus.cc:695] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [term 1 LEADER]: Becoming Leader. State: Replica: 13daf01be9154f5a8303269075497a0b, State: Running, Role: LEADER
I20250812 14:00:41.510354 11624 consensus_queue.cc:237] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } }
I20250812 14:00:41.518779 11076 catalog_manager.cc:5582] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 13daf01be9154f5a8303269075497a0b (127.4.158.2). New cstate: current_term: 1 leader_uuid: "13daf01be9154f5a8303269075497a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13daf01be9154f5a8303269075497a0b" member_type: VOTER last_known_addr { host: "127.4.158.2" port: 43103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 } health_report { overall_health: UNKNOWN } } }
W20250812 14:00:41.565373 11542 tablet.cc:2378] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 14:00:41.593318 11265 tablet.cc:2378] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250812 14:00:41.601188 11403 tablet.cc:2378] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250812 14:00:41.727228 11491 raft_consensus.cc:1273] T 5aefe60d220349e88d56f0342f9b238e P 2f184ca7f701451691bdd85c13b4019b [term 1 FOLLOWER]: Refusing update from remote peer 13daf01be9154f5a8303269075497a0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 14:00:41.728861 11624 consensus_queue.cc:1035] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f184ca7f701451691bdd85c13b4019b" member_type: VOTER last_known_addr { host: "127.4.158.3" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 14:00:41.733399 11219 raft_consensus.cc:1273] T 5aefe60d220349e88d56f0342f9b238e P aec70b8ae82f49f18b52c8e3e81f8750 [term 1 FOLLOWER]: Refusing update from remote peer 13daf01be9154f5a8303269075497a0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250812 14:00:41.735991 11624 consensus_queue.cc:1035] T 5aefe60d220349e88d56f0342f9b238e P 13daf01be9154f5a8303269075497a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec70b8ae82f49f18b52c8e3e81f8750" member_type: VOTER last_known_addr { host: "127.4.158.1" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250812 14:00:41.758481 11639 mvcc.cc:204] Tried to move back new op lower bound from 7188509662101606400 to 7188509661234655232. Current Snapshot: MvccSnapshot[applied={T|T < 7188509662101606400}]
W20250812 14:00:43.350198 11611 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.4.158.2:43103, user_credentials={real_user=slave}} blocked reactor thread for 47734.5us
I20250812 14:00:46.688203 11334 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250812 14:00:46.690686 11471 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250812 14:00:46.691283 11199 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
38b98ed424c04929acc43a674d86e1e5 | 127.4.158.62:33535 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.4.158.20:40709 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
13daf01be9154f5a8303269075497a0b | 127.4.158.2:43103 | HEALTHY | <none> | 1 | 0
2f184ca7f701451691bdd85c13b4019b | 127.4.158.3:44787 | HEALTHY | <none> | 0 | 0
aec70b8ae82f49f18b52c8e3e81f8750 | 127.4.158.1:35683 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.4.158.1 | experimental | 127.4.158.1:35683
local_ip_for_outbound_sockets | 127.4.158.2 | experimental | 127.4.158.2:43103
local_ip_for_outbound_sockets | 127.4.158.3 | experimental | 127.4.158.3:44787
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-0/data/info.pb | hidden | 127.4.158.1:35683
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-1/data/info.pb | hidden | 127.4.158.2:43103
server_dump_info_path | /tmp/dist-test-tasktZztFx/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755007066725366-4728-0/minicluster-data/ts-2/data/info.pb | hidden | 127.4.158.3:44787
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.4.158.20:40709 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250812 14:00:46.891350 4728 log_verifier.cc:126] Checking tablet 4328770079f244428d897a06b135abf7
I20250812 14:00:46.891772 4728 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 4328770079f244428d897a06b135abf7
I20250812 14:00:46.891927 4728 log_verifier.cc:126] Checking tablet 5aefe60d220349e88d56f0342f9b238e
I20250812 14:00:47.667181 4728 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 5aefe60d220349e88d56f0342f9b238e
I20250812 14:00:47.693085 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 11114
I20250812 14:00:47.729225 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 11268
I20250812 14:00:47.762569 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 11402
I20250812 14:00:47.792969 4728 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasktZztFx/build/tsan/bin/kudu with pid 11044
2025-08-12T14:00:47Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35532 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35532 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (181054 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST