Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250815 07:57:19.177914 1511 test_util.cc:276] Using random seed: -1120515009
W20250815 07:57:20.276490 1511 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.053s user 0.353s sys 0.699s
W20250815 07:57:20.276958 1511 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.053s user 0.353s sys 0.699s
I20250815 07:57:20.279079 1511 ts_itest-base.cc:115] Starting cluster with:
I20250815 07:57:20.279302 1511 ts_itest-base.cc:116] --------------
I20250815 07:57:20.279482 1511 ts_itest-base.cc:117] 4 tablet servers
I20250815 07:57:20.279675 1511 ts_itest-base.cc:118] 3 replicas per TS
I20250815 07:57:20.279843 1511 ts_itest-base.cc:119] --------------
2025-08-15T07:57:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T07:57:20Z Disabled control of system clock
I20250815 07:57:20.319506 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:41843
--webserver_interface=127.1.121.254
--webserver_port=0
--builtin_ntp_servers=127.1.121.212:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:41843 with env {}
W20250815 07:57:20.579044 1525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:20.579598 1525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:20.580096 1525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:20.607416 1525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:57:20.607671 1525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:20.607898 1525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:57:20.608119 1525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:57:20.639766 1525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:41843
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:20.641052 1525 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:20.642438 1525 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:20.653046 1531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:20.653537 1532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:21.726706 1534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:21.730482 1533 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1076 milliseconds
I20250815 07:57:21.730551 1525 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:21.731701 1525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:21.734537 1525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:21.735949 1525 hybrid_clock.cc:648] HybridClock initialized: now 1755244641735913 us; error 43 us; skew 500 ppm
I20250815 07:57:21.736758 1525 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:21.742552 1525 webserver.cc:480] Webserver started at http://127.1.121.254:42397/ using document root <none> and password file <none>
I20250815 07:57:21.743355 1525 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:21.743558 1525 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:21.743963 1525 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:21.747974 1525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3e8d74cc8f574896ba4ae034add1caaf"
format_stamp: "Formatted at 2025-08-15 07:57:21 on dist-test-slave-dlgj"
I20250815 07:57:21.749009 1525 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3e8d74cc8f574896ba4ae034add1caaf"
format_stamp: "Formatted at 2025-08-15 07:57:21 on dist-test-slave-dlgj"
I20250815 07:57:21.755270 1525 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250815 07:57:21.760196 1541 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:21.761128 1525 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250815 07:57:21.761412 1525 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3e8d74cc8f574896ba4ae034add1caaf"
format_stamp: "Formatted at 2025-08-15 07:57:21 on dist-test-slave-dlgj"
I20250815 07:57:21.761693 1525 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:21.812314 1525 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:21.813592 1525 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:21.813975 1525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:21.878742 1525 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:41843
I20250815 07:57:21.878798 1592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:41843 every 8 connection(s)
I20250815 07:57:21.881276 1525 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:57:21.882611 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 1525
I20250815 07:57:21.882992 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 07:57:21.886987 1593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:21.903990 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Bootstrap starting.
I20250815 07:57:21.908851 1593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:21.910356 1593 log.cc:826] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:21.914741 1593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: No bootstrap required, opened a new log
I20250815 07:57:21.931829 1593 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } }
I20250815 07:57:21.932504 1593 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:21.932775 1593 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8d74cc8f574896ba4ae034add1caaf, State: Initialized, Role: FOLLOWER
I20250815 07:57:21.933354 1593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } }
I20250815 07:57:21.933790 1593 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:57:21.934021 1593 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:57:21.934293 1593 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:21.938181 1593 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } }
I20250815 07:57:21.938805 1593 leader_election.cc:304] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e8d74cc8f574896ba4ae034add1caaf; no voters:
I20250815 07:57:21.940281 1593 leader_election.cc:290] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:57:21.941002 1598 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:57:21.942968 1598 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 1 LEADER]: Becoming Leader. State: Replica: 3e8d74cc8f574896ba4ae034add1caaf, State: Running, Role: LEADER
I20250815 07:57:21.943579 1598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } }
I20250815 07:57:21.944479 1593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:57:21.953305 1599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e8d74cc8f574896ba4ae034add1caaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } } }
I20250815 07:57:21.954239 1600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8d74cc8f574896ba4ae034add1caaf. Latest consensus state: current_term: 1 leader_uuid: "3e8d74cc8f574896ba4ae034add1caaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } } }
I20250815 07:57:21.954602 1599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: This master's current role is: LEADER
I20250815 07:57:21.954851 1600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: This master's current role is: LEADER
I20250815 07:57:21.956929 1608 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:57:21.966784 1608 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:57:21.982053 1608 catalog_manager.cc:1349] Generated new cluster ID: a6e04818011549b08d33c471df9dd042
I20250815 07:57:21.982293 1608 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:57:22.012122 1608 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 07:57:22.013350 1608 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:57:22.033793 1608 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Generated new TSK 0
I20250815 07:57:22.034649 1608 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:57:22.057119 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:0
--local_ip_for_outbound_sockets=127.1.121.193
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:41843
--builtin_ntp_servers=127.1.121.212:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 07:57:22.347795 1617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:22.348268 1617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:22.348717 1617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:22.376797 1617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:22.377522 1617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:57:22.407830 1617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:41843
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:22.409030 1617 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:22.410493 1617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:22.421706 1623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:22.422979 1624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:23.752552 1617 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.329s user 0.001s sys 0.007s
W20250815 07:57:23.752990 1617 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.329s user 0.001s sys 0.008s
W20250815 07:57:23.758631 1626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:23.758951 1617 server_base.cc:1047] running on GCE node
I20250815 07:57:23.760578 1617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:23.763168 1617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:23.764638 1617 hybrid_clock.cc:648] HybridClock initialized: now 1755244643764590 us; error 36 us; skew 500 ppm
I20250815 07:57:23.765692 1617 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:23.773784 1617 webserver.cc:480] Webserver started at http://127.1.121.193:38075/ using document root <none> and password file <none>
I20250815 07:57:23.774982 1617 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:23.775242 1617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:23.775817 1617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:23.781692 1617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bf8333e9ff054a62936a869121c2eda2"
format_stamp: "Formatted at 2025-08-15 07:57:23 on dist-test-slave-dlgj"
I20250815 07:57:23.783167 1617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bf8333e9ff054a62936a869121c2eda2"
format_stamp: "Formatted at 2025-08-15 07:57:23 on dist-test-slave-dlgj"
I20250815 07:57:23.791800 1617 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250815 07:57:23.798864 1633 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:23.800112 1617 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20250815 07:57:23.800559 1617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bf8333e9ff054a62936a869121c2eda2"
format_stamp: "Formatted at 2025-08-15 07:57:23 on dist-test-slave-dlgj"
I20250815 07:57:23.801023 1617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:23.895727 1617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:23.897531 1617 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:23.897997 1617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:23.901005 1617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:23.906169 1617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:23.906414 1617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:23.906704 1617 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:23.906929 1617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:24.050257 1617 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:39201
I20250815 07:57:24.050318 1745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:39201 every 8 connection(s)
I20250815 07:57:24.053757 1617 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:57:24.058308 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 1617
I20250815 07:57:24.058770 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 07:57:24.068316 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:0
--local_ip_for_outbound_sockets=127.1.121.194
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:41843
--builtin_ntp_servers=127.1.121.212:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:57:24.078197 1746 heartbeater.cc:344] Connected to a master server at 127.1.121.254:41843
I20250815 07:57:24.078662 1746 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:24.079807 1746 heartbeater.cc:507] Master 127.1.121.254:41843 requested a full tablet report, sending...
I20250815 07:57:24.082665 1558 ts_manager.cc:194] Registered new tserver with Master: bf8333e9ff054a62936a869121c2eda2 (127.1.121.193:39201)
I20250815 07:57:24.085319 1558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:60177
W20250815 07:57:24.355333 1750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:24.355793 1750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:24.356321 1750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:24.385521 1750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:24.386355 1750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:57:24.416803 1750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:41843
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:24.417985 1750 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:24.419461 1750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:24.430125 1756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:25.088910 1746 heartbeater.cc:499] Master 127.1.121.254:41843 was elected leader, sending a full tablet report...
W20250815 07:57:25.833802 1755 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 1750
W20250815 07:57:25.990825 1750 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.560s user 0.668s sys 0.891s
W20250815 07:57:24.430680 1757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:25.991189 1750 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.561s user 0.669s sys 0.892s
W20250815 07:57:25.993316 1759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:25.995699 1758 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1560 milliseconds
I20250815 07:57:25.995718 1750 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:25.996878 1750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:25.998713 1750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:26.000048 1750 hybrid_clock.cc:648] HybridClock initialized: now 1755244645999988 us; error 39 us; skew 500 ppm
I20250815 07:57:26.000757 1750 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:26.006248 1750 webserver.cc:480] Webserver started at http://127.1.121.194:41131/ using document root <none> and password file <none>
I20250815 07:57:26.007050 1750 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:26.007258 1750 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:26.007758 1750 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:26.011652 1750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fcfe8eecaf124ec9bfe962a883a5459b"
format_stamp: "Formatted at 2025-08-15 07:57:26 on dist-test-slave-dlgj"
I20250815 07:57:26.012634 1750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fcfe8eecaf124ec9bfe962a883a5459b"
format_stamp: "Formatted at 2025-08-15 07:57:26 on dist-test-slave-dlgj"
I20250815 07:57:26.019006 1750 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250815 07:57:26.023897 1766 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:26.024803 1750 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250815 07:57:26.025092 1750 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fcfe8eecaf124ec9bfe962a883a5459b"
format_stamp: "Formatted at 2025-08-15 07:57:26 on dist-test-slave-dlgj"
I20250815 07:57:26.025398 1750 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:26.088366 1750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:26.090333 1750 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:26.090924 1750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:26.094354 1750 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:26.100464 1750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:26.100772 1750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:26.101155 1750 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:26.101402 1750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:26.224105 1750 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:44545
I20250815 07:57:26.224185 1878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:44545 every 8 connection(s)
I20250815 07:57:26.226430 1750 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 07:57:26.236552 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 1750
I20250815 07:57:26.236892 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 07:57:26.241848 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:0
--local_ip_for_outbound_sockets=127.1.121.195
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:41843
--builtin_ntp_servers=127.1.121.212:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:57:26.243870 1879 heartbeater.cc:344] Connected to a master server at 127.1.121.254:41843
I20250815 07:57:26.244230 1879 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:26.245110 1879 heartbeater.cc:507] Master 127.1.121.254:41843 requested a full tablet report, sending...
I20250815 07:57:26.247226 1558 ts_manager.cc:194] Registered new tserver with Master: fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545)
I20250815 07:57:26.248642 1558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:52619
W20250815 07:57:26.510033 1883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:26.510447 1883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:26.510830 1883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:26.537767 1883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:26.538414 1883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:57:26.567705 1883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:41843
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:26.568782 1883 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:26.570094 1883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:26.580950 1889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:27.251369 1879 heartbeater.cc:499] Master 127.1.121.254:41843 was elected leader, sending a full tablet report...
W20250815 07:57:27.984977 1888 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 1883
W20250815 07:57:28.010344 1883 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.430s user 0.526s sys 0.890s
W20250815 07:57:26.581307 1890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:28.010763 1883 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.430s user 0.526s sys 0.891s
W20250815 07:57:28.012277 1892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:28.014793 1891 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1430 milliseconds
I20250815 07:57:28.014807 1883 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:28.015836 1883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:28.017604 1883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:28.018949 1883 hybrid_clock.cc:648] HybridClock initialized: now 1755244648018912 us; error 37 us; skew 500 ppm
I20250815 07:57:28.019609 1883 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:28.024863 1883 webserver.cc:480] Webserver started at http://127.1.121.195:33439/ using document root <none> and password file <none>
I20250815 07:57:28.025672 1883 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:28.025869 1883 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:28.026257 1883 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:28.030086 1883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "87807a99b0e24018b7083be5306e7797"
format_stamp: "Formatted at 2025-08-15 07:57:28 on dist-test-slave-dlgj"
I20250815 07:57:28.031006 1883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "87807a99b0e24018b7083be5306e7797"
format_stamp: "Formatted at 2025-08-15 07:57:28 on dist-test-slave-dlgj"
I20250815 07:57:28.037359 1883 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250815 07:57:28.042133 1899 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:28.043023 1883 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250815 07:57:28.043284 1883 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "87807a99b0e24018b7083be5306e7797"
format_stamp: "Formatted at 2025-08-15 07:57:28 on dist-test-slave-dlgj"
I20250815 07:57:28.043560 1883 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:28.095417 1883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:28.096843 1883 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:28.097234 1883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:28.099539 1883 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:28.103765 1883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:28.104032 1883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:28.104354 1883 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:28.104535 1883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:28.230217 1883 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:38903
I20250815 07:57:28.230370 2011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:38903 every 8 connection(s)
I20250815 07:57:28.232743 1883 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 07:57:28.242002 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 1883
I20250815 07:57:28.242349 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 07:57:28.248158 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.196:0
--local_ip_for_outbound_sockets=127.1.121.196
--webserver_interface=127.1.121.196
--webserver_port=0
--tserver_master_addrs=127.1.121.254:41843
--builtin_ntp_servers=127.1.121.212:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:57:28.255719 2012 heartbeater.cc:344] Connected to a master server at 127.1.121.254:41843
I20250815 07:57:28.256171 2012 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:28.257460 2012 heartbeater.cc:507] Master 127.1.121.254:41843 requested a full tablet report, sending...
I20250815 07:57:28.259485 1558 ts_manager.cc:194] Registered new tserver with Master: 87807a99b0e24018b7083be5306e7797 (127.1.121.195:38903)
I20250815 07:57:28.261080 1558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:37869
W20250815 07:57:28.510188 2016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:28.510564 2016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:28.510931 2016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:28.537128 2016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:28.537804 2016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.196
I20250815 07:57:28.566754 2016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.1.121.196
--webserver_port=0
--tserver_master_addrs=127.1.121.254:41843
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.196
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:28.567888 2016 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:28.569240 2016 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:28.580071 2022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:29.263866 2012 heartbeater.cc:499] Master 127.1.121.254:41843 was elected leader, sending a full tablet report...
W20250815 07:57:28.580879 2023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:29.630743 2025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:29.632611 2024 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1048 milliseconds
I20250815 07:57:29.632687 2016 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:29.633700 2016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:29.636175 2016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:29.637580 2016 hybrid_clock.cc:648] HybridClock initialized: now 1755244649637535 us; error 51 us; skew 500 ppm
I20250815 07:57:29.638216 2016 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:29.643314 2016 webserver.cc:480] Webserver started at http://127.1.121.196:41069/ using document root <none> and password file <none>
I20250815 07:57:29.644140 2016 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:29.644309 2016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:29.644685 2016 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:29.648577 2016 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "688aefe7bb8f4cf393dd157b5113d217"
format_stamp: "Formatted at 2025-08-15 07:57:29 on dist-test-slave-dlgj"
I20250815 07:57:29.649456 2016 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "688aefe7bb8f4cf393dd157b5113d217"
format_stamp: "Formatted at 2025-08-15 07:57:29 on dist-test-slave-dlgj"
I20250815 07:57:29.655321 2016 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250815 07:57:29.660077 2033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:29.660962 2016 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250815 07:57:29.661229 2016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "688aefe7bb8f4cf393dd157b5113d217"
format_stamp: "Formatted at 2025-08-15 07:57:29 on dist-test-slave-dlgj"
I20250815 07:57:29.661512 2016 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:29.712177 2016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:29.713383 2016 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:29.713762 2016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:29.715924 2016 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:29.719267 2016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:29.719444 2016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:29.719705 2016 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:29.719836 2016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:29.835621 2016 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.196:46435
I20250815 07:57:29.835714 2145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.196:46435 every 8 connection(s)
I20250815 07:57:29.838027 2016 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250815 07:57:29.848152 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 2016
I20250815 07:57:29.848639 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250815 07:57:29.856278 2146 heartbeater.cc:344] Connected to a master server at 127.1.121.254:41843
I20250815 07:57:29.856660 2146 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:29.857554 2146 heartbeater.cc:507] Master 127.1.121.254:41843 requested a full tablet report, sending...
I20250815 07:57:29.859139 1558 ts_manager.cc:194] Registered new tserver with Master: 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196:46435)
I20250815 07:57:29.860208 1558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.196:51015
I20250815 07:57:29.867340 1511 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250815 07:57:29.905194 1558 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60090:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 07:57:29.967876 2081 tablet_service.cc:1468] Processing CreateTablet for tablet ae238bcff75a4bc9952d37cc186c5b3c (DEFAULT_TABLE table=TestTable [id=a41e26d30a284623822a917eb2d6cdc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:29.969604 2081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae238bcff75a4bc9952d37cc186c5b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:29.969522 1947 tablet_service.cc:1468] Processing CreateTablet for tablet ae238bcff75a4bc9952d37cc186c5b3c (DEFAULT_TABLE table=TestTable [id=a41e26d30a284623822a917eb2d6cdc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:29.971092 1947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae238bcff75a4bc9952d37cc186c5b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:29.975349 1814 tablet_service.cc:1468] Processing CreateTablet for tablet ae238bcff75a4bc9952d37cc186c5b3c (DEFAULT_TABLE table=TestTable [id=a41e26d30a284623822a917eb2d6cdc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:29.977197 1814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae238bcff75a4bc9952d37cc186c5b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:29.990070 2165 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: Bootstrap starting.
I20250815 07:57:29.994014 2166 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217: Bootstrap starting.
I20250815 07:57:29.997272 2165 tablet_bootstrap.cc:654] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:29.999440 2165 log.cc:826] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:30.000977 2166 tablet_bootstrap.cc:654] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:30.001065 2167 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b: Bootstrap starting.
I20250815 07:57:30.002925 2166 log.cc:826] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:30.005053 2165 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: No bootstrap required, opened a new log
I20250815 07:57:30.005568 2165 ts_tablet_manager.cc:1397] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250815 07:57:30.007493 2166 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217: No bootstrap required, opened a new log
I20250815 07:57:30.007874 2166 ts_tablet_manager.cc:1397] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20250815 07:57:30.007921 2167 tablet_bootstrap.cc:654] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:30.009935 2167 log.cc:826] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:30.014755 2167 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b: No bootstrap required, opened a new log
I20250815 07:57:30.015158 2167 ts_tablet_manager.cc:1397] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20250815 07:57:30.024583 2166 raft_consensus.cc:357] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.025178 2166 raft_consensus.cc:383] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:30.025424 2166 raft_consensus.cc:738] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688aefe7bb8f4cf393dd157b5113d217, State: Initialized, Role: FOLLOWER
I20250815 07:57:30.026093 2166 consensus_queue.cc:260] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.029206 2146 heartbeater.cc:499] Master 127.1.121.254:41843 was elected leader, sending a full tablet report...
I20250815 07:57:30.028999 2165 raft_consensus.cc:357] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.029850 2165 raft_consensus.cc:383] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:30.030149 2165 raft_consensus.cc:738] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87807a99b0e24018b7083be5306e7797, State: Initialized, Role: FOLLOWER
I20250815 07:57:30.030443 2166 ts_tablet_manager.cc:1428] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20250815 07:57:30.031054 2165 consensus_queue.cc:260] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.034634 2165 ts_tablet_manager.cc:1428] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: Time spent starting tablet: real 0.029s user 0.027s sys 0.000s
I20250815 07:57:30.039481 2167 raft_consensus.cc:357] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.040161 2167 raft_consensus.cc:383] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:30.040486 2167 raft_consensus.cc:738] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfe8eecaf124ec9bfe962a883a5459b, State: Initialized, Role: FOLLOWER
I20250815 07:57:30.041265 2167 consensus_queue.cc:260] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.043987 2167 ts_tablet_manager.cc:1428] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b: Time spent starting tablet: real 0.029s user 0.026s sys 0.003s
W20250815 07:57:30.092237 2147 tablet.cc:2378] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:57:30.157011 2173 raft_consensus.cc:491] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:57:30.157475 2173 raft_consensus.cc:513] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.159837 2173 leader_election.cc:290] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196:46435), 87807a99b0e24018b7083be5306e7797 (127.1.121.195:38903)
I20250815 07:57:30.171528 2101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688aefe7bb8f4cf393dd157b5113d217" is_pre_election: true
I20250815 07:57:30.172204 2101 raft_consensus.cc:2466] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfe8eecaf124ec9bfe962a883a5459b in term 0.
I20250815 07:57:30.172385 1967 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87807a99b0e24018b7083be5306e7797" is_pre_election: true
I20250815 07:57:30.173094 1967 raft_consensus.cc:2466] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfe8eecaf124ec9bfe962a883a5459b in term 0.
I20250815 07:57:30.173239 1769 leader_election.cc:304] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 688aefe7bb8f4cf393dd157b5113d217, fcfe8eecaf124ec9bfe962a883a5459b; no voters:
I20250815 07:57:30.174019 2173 raft_consensus.cc:2802] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 07:57:30.174275 2173 raft_consensus.cc:491] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 07:57:30.174510 2173 raft_consensus.cc:3058] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:30.178539 2173 raft_consensus.cc:513] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.179888 2173 leader_election.cc:290] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [CANDIDATE]: Term 1 election: Requested vote from peers 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196:46435), 87807a99b0e24018b7083be5306e7797 (127.1.121.195:38903)
I20250815 07:57:30.180492 2101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "688aefe7bb8f4cf393dd157b5113d217"
I20250815 07:57:30.180855 2101 raft_consensus.cc:3058] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:30.180718 1967 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87807a99b0e24018b7083be5306e7797"
I20250815 07:57:30.181185 1967 raft_consensus.cc:3058] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:30.185122 2101 raft_consensus.cc:2466] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfe8eecaf124ec9bfe962a883a5459b in term 1.
I20250815 07:57:30.185256 1967 raft_consensus.cc:2466] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfe8eecaf124ec9bfe962a883a5459b in term 1.
I20250815 07:57:30.185912 1769 leader_election.cc:304] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 688aefe7bb8f4cf393dd157b5113d217, fcfe8eecaf124ec9bfe962a883a5459b; no voters:
I20250815 07:57:30.186535 2173 raft_consensus.cc:2802] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:57:30.187880 2173 raft_consensus.cc:695] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [term 1 LEADER]: Becoming Leader. State: Replica: fcfe8eecaf124ec9bfe962a883a5459b, State: Running, Role: LEADER
I20250815 07:57:30.188530 2173 consensus_queue.cc:237] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:30.197605 1557 catalog_manager.cc:5582] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b reported cstate change: term changed from 0 to 1, leader changed from <none> to fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194). New cstate: current_term: 1 leader_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } health_report { overall_health: HEALTHY } } }
I20250815 07:57:30.216176 1511 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250815 07:57:30.219288 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fcfe8eecaf124ec9bfe962a883a5459b to finish bootstrapping
I20250815 07:57:30.231880 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 87807a99b0e24018b7083be5306e7797 to finish bootstrapping
W20250815 07:57:30.234427 1880 tablet.cc:2378] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 07:57:30.238492 2013 tablet.cc:2378] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:57:30.242321 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 688aefe7bb8f4cf393dd157b5113d217 to finish bootstrapping
I20250815 07:57:30.252214 1511 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250815 07:57:30.255012 1511 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217"
rpc_addresses {
host: "127.1.121.196"
port: 46435
}
}
ts_infos {
permanent_uuid: "87807a99b0e24018b7083be5306e7797"
rpc_addresses {
host: "127.1.121.195"
port: 38903
}
}
ts_infos {
permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b"
rpc_addresses {
host: "127.1.121.194"
port: 44545
}
}
I20250815 07:57:30.630437 2173 consensus_queue.cc:1035] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [LEADER]: Connected to new peer: Peer: permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 07:57:30.643878 2173 consensus_queue.cc:1035] T ae238bcff75a4bc9952d37cc186c5b3c P fcfe8eecaf124ec9bfe962a883a5459b [LEADER]: Connected to new peer: Peer: permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 07:57:30.644461 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 1750
W20250815 07:57:30.667264 1900 connection.cc:537] server connection from 127.1.121.194:54783 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250815 07:57:30.668176 1544 connection.cc:537] server connection from 127.1.121.194:52619 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250815 07:57:30.668452 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 1525
I20250815 07:57:30.692328 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:41843
--webserver_interface=127.1.121.254
--webserver_port=42397
--builtin_ntp_servers=127.1.121.212:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:41843 with env {}
W20250815 07:57:30.971257 2185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:30.971750 2185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:30.972189 2185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:30.998169 2185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:57:30.998411 2185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:30.998673 2185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:57:30.998899 2185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:57:31.029454 2185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:41843
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:41843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=42397
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:31.030519 2185 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:31.031829 2185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:31.041183 2191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:31.045835 2146 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:41843 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:41843: connect: Connection refused (error 111)
W20250815 07:57:31.126967 1746 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:41843 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:41843: connect: Connection refused (error 111)
W20250815 07:57:31.669796 2012 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:41843 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:41843: connect: Connection refused (error 111)
I20250815 07:57:31.938097 2200 raft_consensus.cc:491] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:57:31.938881 2200 raft_consensus.cc:513] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:31.948812 2200 leader_election.cc:290] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87807a99b0e24018b7083be5306e7797 (127.1.121.195:38903), fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545)
W20250815 07:57:31.960564 2035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.194:44545: connect: Connection refused (error 111)
I20250815 07:57:31.966620 1967 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "688aefe7bb8f4cf393dd157b5113d217" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87807a99b0e24018b7083be5306e7797" is_pre_election: true
W20250815 07:57:31.968009 2035 leader_election.cc:336] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545): Network error: Client connection negotiation failed: client connection to 127.1.121.194:44545: connect: Connection refused (error 111)
I20250815 07:57:31.969179 2035 leader_election.cc:304] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 688aefe7bb8f4cf393dd157b5113d217; no voters: 87807a99b0e24018b7083be5306e7797, fcfe8eecaf124ec9bfe962a883a5459b
I20250815 07:57:31.970134 2200 raft_consensus.cc:2747] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250815 07:57:32.150768 2206 raft_consensus.cc:491] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fcfe8eecaf124ec9bfe962a883a5459b)
I20250815 07:57:32.151530 2206 raft_consensus.cc:513] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:32.156631 2206 leader_election.cc:290] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196:46435), fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545)
W20250815 07:57:32.161945 1901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.194:44545: connect: Connection refused (error 111)
W20250815 07:57:32.176659 1901 leader_election.cc:336] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545): Network error: Client connection negotiation failed: client connection to 127.1.121.194:44545: connect: Connection refused (error 111)
I20250815 07:57:32.191874 2101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "87807a99b0e24018b7083be5306e7797" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "688aefe7bb8f4cf393dd157b5113d217" is_pre_election: true
I20250815 07:57:32.192677 2101 raft_consensus.cc:2466] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87807a99b0e24018b7083be5306e7797 in term 1.
I20250815 07:57:32.194478 1902 leader_election.cc:304] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 688aefe7bb8f4cf393dd157b5113d217, 87807a99b0e24018b7083be5306e7797; no voters: fcfe8eecaf124ec9bfe962a883a5459b
I20250815 07:57:32.195896 2206 raft_consensus.cc:2802] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 07:57:32.196310 2206 raft_consensus.cc:491] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fcfe8eecaf124ec9bfe962a883a5459b)
I20250815 07:57:32.196651 2206 raft_consensus.cc:3058] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 1 FOLLOWER]: Advancing to term 2
I20250815 07:57:32.205621 2206 raft_consensus.cc:513] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:32.209204 2101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "87807a99b0e24018b7083be5306e7797" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "688aefe7bb8f4cf393dd157b5113d217"
I20250815 07:57:32.209867 2101 raft_consensus.cc:3058] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 1 FOLLOWER]: Advancing to term 2
W20250815 07:57:32.230967 1901 leader_election.cc:336] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545): Network error: Client connection negotiation failed: client connection to 127.1.121.194:44545: connect: Connection refused (error 111)
I20250815 07:57:32.238890 2206 leader_election.cc:290] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 2 election: Requested vote from peers 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196:46435), fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545)
I20250815 07:57:32.218711 2101 raft_consensus.cc:2466] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87807a99b0e24018b7083be5306e7797 in term 2.
I20250815 07:57:32.240442 1902 leader_election.cc:304] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 688aefe7bb8f4cf393dd157b5113d217, 87807a99b0e24018b7083be5306e7797; no voters: fcfe8eecaf124ec9bfe962a883a5459b
I20250815 07:57:32.243455 2210 raft_consensus.cc:2802] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 07:57:32.244930 2210 raft_consensus.cc:695] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 2 LEADER]: Becoming Leader. State: Replica: 87807a99b0e24018b7083be5306e7797, State: Running, Role: LEADER
I20250815 07:57:32.246306 2210 consensus_queue.cc:237] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:32.632555 2101 raft_consensus.cc:1273] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 2 FOLLOWER]: Refusing update from remote peer 87807a99b0e24018b7083be5306e7797: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 07:57:32.634989 2210 consensus_queue.cc:1035] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 07:57:32.443616 2190 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 2185
W20250815 07:57:32.665419 2190 kernel_stack_watchdog.cc:198] Thread 2185 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 07:57:32.665545 2185 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.623s user 0.561s sys 1.018s
W20250815 07:57:31.041675 2192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:32.665902 2185 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.624s user 0.561s sys 1.019s
W20250815 07:57:32.667829 2194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:32.672441 2193 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1626 milliseconds
I20250815 07:57:32.672470 2185 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:32.673926 2185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:32.686996 2185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:32.688520 2185 hybrid_clock.cc:648] HybridClock initialized: now 1755244652688478 us; error 40 us; skew 500 ppm
I20250815 07:57:32.689502 2185 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250815 07:57:32.689728 1901 consensus_peers.cc:489] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 -> Peer fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545): Couldn't send request to peer fcfe8eecaf124ec9bfe962a883a5459b. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.194:44545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 07:57:32.695269 2185 webserver.cc:480] Webserver started at http://127.1.121.254:42397/ using document root <none> and password file <none>
I20250815 07:57:32.696159 2185 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:32.696359 2185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:32.703346 2185 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20250815 07:57:32.707545 2224 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:32.708472 2185 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250815 07:57:32.708755 2185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3e8d74cc8f574896ba4ae034add1caaf"
format_stamp: "Formatted at 2025-08-15 07:57:21 on dist-test-slave-dlgj"
I20250815 07:57:32.710493 2185 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:32.780122 2185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:32.781466 2185 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:32.781836 2185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:32.845640 2185 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:41843
I20250815 07:57:32.845686 2275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:41843 every 8 connection(s)
I20250815 07:57:32.849064 2185 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:57:32.854059 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 2185
I20250815 07:57:32.854624 1511 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 87807a99b0e24018b7083be5306e7797
I20250815 07:57:32.861646 2276 sys_catalog.cc:263] Verifying existing consensus state
I20250815 07:57:32.866524 2276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Bootstrap starting.
I20250815 07:57:32.901986 2276 log.cc:826] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:32.922672 2276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:57:32.923386 2276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Bootstrap complete.
I20250815 07:57:32.943017 2276 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } }
I20250815 07:57:32.944934 2276 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8d74cc8f574896ba4ae034add1caaf, State: Initialized, Role: FOLLOWER
I20250815 07:57:32.945681 2276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } }
I20250815 07:57:32.946133 2276 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:57:32.946372 2276 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:57:32.946642 2276 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 1 FOLLOWER]: Advancing to term 2
I20250815 07:57:32.951675 2276 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } }
I20250815 07:57:32.952307 2276 leader_election.cc:304] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e8d74cc8f574896ba4ae034add1caaf; no voters:
I20250815 07:57:32.954236 2276 leader_election.cc:290] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 07:57:32.954725 2280 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 2 FOLLOWER]: Leader election won for term 2
I20250815 07:57:32.957648 2280 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [term 2 LEADER]: Becoming Leader. State: Replica: 3e8d74cc8f574896ba4ae034add1caaf, State: Running, Role: LEADER
I20250815 07:57:32.958402 2280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } }
I20250815 07:57:32.959070 2276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:57:32.966849 2281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3e8d74cc8f574896ba4ae034add1caaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } } }
I20250815 07:57:32.967890 2282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e8d74cc8f574896ba4ae034add1caaf. Latest consensus state: current_term: 2 leader_uuid: "3e8d74cc8f574896ba4ae034add1caaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e8d74cc8f574896ba4ae034add1caaf" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 41843 } } }
I20250815 07:57:32.968465 2282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: This master's current role is: LEADER
I20250815 07:57:32.970536 2281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf [sys.catalog]: This master's current role is: LEADER
I20250815 07:57:32.974198 2286 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:57:32.988847 2286 catalog_manager.cc:671] Loaded metadata for table TestTable [id=a41e26d30a284623822a917eb2d6cdc3]
I20250815 07:57:32.997275 2286 tablet_loader.cc:96] loaded metadata for tablet ae238bcff75a4bc9952d37cc186c5b3c (table TestTable [id=a41e26d30a284623822a917eb2d6cdc3])
I20250815 07:57:32.998549 2286 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:57:33.003636 2286 catalog_manager.cc:1261] Loaded cluster ID: a6e04818011549b08d33c471df9dd042
I20250815 07:57:33.003986 2286 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:57:33.012287 2286 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:57:33.017261 2286 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 3e8d74cc8f574896ba4ae034add1caaf: Loaded TSK: 0
I20250815 07:57:33.018864 2286 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:57:33.177691 1746 heartbeater.cc:344] Connected to a master server at 127.1.121.254:41843
W20250815 07:57:33.182828 2278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:33.183384 2278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250815 07:57:33.183390 2241 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" instance_seqno: 1755244644017362) as {username='slave'} at 127.1.121.193:56857; Asking this server to re-register.
I20250815 07:57:33.184836 1746 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:33.185454 1746 heartbeater.cc:507] Master 127.1.121.254:41843 requested a full tablet report, sending...
I20250815 07:57:33.187610 2241 ts_manager.cc:194] Registered new tserver with Master: bf8333e9ff054a62936a869121c2eda2 (127.1.121.193:39201)
W20250815 07:57:33.214248 2278 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250815 07:57:33.710063 2012 heartbeater.cc:344] Connected to a master server at 127.1.121.254:41843
I20250815 07:57:33.713572 2241 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "87807a99b0e24018b7083be5306e7797" instance_seqno: 1755244648200365) as {username='slave'} at 127.1.121.195:60541; Asking this server to re-register.
I20250815 07:57:33.714741 2012 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:33.715234 2012 heartbeater.cc:507] Master 127.1.121.254:41843 requested a full tablet report, sending...
I20250815 07:57:33.718156 2241 ts_manager.cc:194] Registered new tserver with Master: 87807a99b0e24018b7083be5306e7797 (127.1.121.195:38903)
I20250815 07:57:33.721728 2241 catalog_manager.cc:5582] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 reported cstate change: term changed from 1 to 2, leader changed from fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194) to 87807a99b0e24018b7083be5306e7797 (127.1.121.195). New cstate: current_term: 2 leader_uuid: "87807a99b0e24018b7083be5306e7797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } health_report { overall_health: UNKNOWN } } }
I20250815 07:57:33.732204 2146 heartbeater.cc:344] Connected to a master server at 127.1.121.254:41843
I20250815 07:57:33.734843 2240 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" instance_seqno: 1755244649808722) as {username='slave'} at 127.1.121.196:45323; Asking this server to re-register.
I20250815 07:57:33.736088 2146 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:33.736671 2146 heartbeater.cc:507] Master 127.1.121.254:41843 requested a full tablet report, sending...
I20250815 07:57:33.739132 2240 ts_manager.cc:194] Registered new tserver with Master: 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196:46435)
W20250815 07:57:34.656746 2304 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 2278
W20250815 07:57:34.657212 2304 kernel_stack_watchdog.cc:198] Thread 2278 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 07:57:34.770805 2278 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.517s user 0.001s sys 0.001s
W20250815 07:57:34.879419 2278 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.625s user 0.005s sys 0.006s
I20250815 07:57:34.917953 1967 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "87807a99b0e24018b7083be5306e7797"
tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217"
}
peers {
permanent_uuid: "87807a99b0e24018b7083be5306e7797"
}
}
from {username='slave'} at 127.0.0.1:59938
W20250815 07:57:34.918807 1967 raft_consensus.cc:2216] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } unsafe_config_change: true
I20250815 07:57:34.919528 1967 raft_consensus.cc:3053] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 2 LEADER]: Stepping down as leader of term 2
I20250815 07:57:34.919741 1967 raft_consensus.cc:738] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 87807a99b0e24018b7083be5306e7797, State: Running, Role: LEADER
I20250815 07:57:34.920302 1967 consensus_queue.cc:260] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:34.921131 1967 raft_consensus.cc:3058] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 2 FOLLOWER]: Advancing to term 3
I20250815 07:57:36.396342 2329 raft_consensus.cc:491] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 87807a99b0e24018b7083be5306e7797)
I20250815 07:57:36.396754 2329 raft_consensus.cc:513] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } }
I20250815 07:57:36.398141 2329 leader_election.cc:290] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 87807a99b0e24018b7083be5306e7797 (127.1.121.195:38903), fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545)
I20250815 07:57:36.398957 1967 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "688aefe7bb8f4cf393dd157b5113d217" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "87807a99b0e24018b7083be5306e7797" is_pre_election: true
W20250815 07:57:36.402328 2035 leader_election.cc:336] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194:44545): Network error: Client connection negotiation failed: client connection to 127.1.121.194:44545: connect: Connection refused (error 111)
I20250815 07:57:36.402576 2035 leader_election.cc:304] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 688aefe7bb8f4cf393dd157b5113d217; no voters: 87807a99b0e24018b7083be5306e7797, fcfe8eecaf124ec9bfe962a883a5459b
I20250815 07:57:36.403003 2329 raft_consensus.cc:2747] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250815 07:57:36.425594 2332 raft_consensus.cc:491] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250815 07:57:36.425926 2332 raft_consensus.cc:513] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } unsafe_config_change: true
I20250815 07:57:36.426786 2332 leader_election.cc:290] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196:46435)
I20250815 07:57:36.427639 2101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "87807a99b0e24018b7083be5306e7797" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "688aefe7bb8f4cf393dd157b5113d217" is_pre_election: true
I20250815 07:57:36.428056 2101 raft_consensus.cc:2466] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87807a99b0e24018b7083be5306e7797 in term 2.
I20250815 07:57:36.428812 1902 leader_election.cc:304] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 688aefe7bb8f4cf393dd157b5113d217, 87807a99b0e24018b7083be5306e7797; no voters:
I20250815 07:57:36.429325 2332 raft_consensus.cc:2802] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250815 07:57:36.429541 2332 raft_consensus.cc:491] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250815 07:57:36.429769 2332 raft_consensus.cc:3058] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 3 FOLLOWER]: Advancing to term 4
I20250815 07:57:36.433467 2332 raft_consensus.cc:513] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } unsafe_config_change: true
I20250815 07:57:36.434336 2332 leader_election.cc:290] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 4 election: Requested vote from peers 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196:46435)
I20250815 07:57:36.435097 2101 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c" candidate_uuid: "87807a99b0e24018b7083be5306e7797" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "688aefe7bb8f4cf393dd157b5113d217"
I20250815 07:57:36.435446 2101 raft_consensus.cc:3058] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 2 FOLLOWER]: Advancing to term 4
I20250815 07:57:36.439000 2101 raft_consensus.cc:2466] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87807a99b0e24018b7083be5306e7797 in term 4.
I20250815 07:57:36.439663 1902 leader_election.cc:304] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 688aefe7bb8f4cf393dd157b5113d217, 87807a99b0e24018b7083be5306e7797; no voters:
I20250815 07:57:36.440202 2332 raft_consensus.cc:2802] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 4 FOLLOWER]: Leader election won for term 4
I20250815 07:57:36.441032 2332 raft_consensus.cc:695] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 4 LEADER]: Becoming Leader. State: Replica: 87807a99b0e24018b7083be5306e7797, State: Running, Role: LEADER
I20250815 07:57:36.441615 2332 consensus_queue.cc:237] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } unsafe_config_change: true
I20250815 07:57:36.446133 2241 catalog_manager.cc:5582] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196), VOTER 87807a99b0e24018b7083be5306e7797 (127.1.121.195). New cstate: current_term: 4 leader_uuid: "87807a99b0e24018b7083be5306e7797" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfe8eecaf124ec9bfe962a883a5459b" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 44545 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } unsafe_config_change: true }
I20250815 07:57:37.016149 2101 raft_consensus.cc:1273] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 4 FOLLOWER]: Refusing update from remote peer 87807a99b0e24018b7083be5306e7797: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250815 07:57:37.017247 2340 consensus_queue.cc:1035] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250815 07:57:37.024737 2343 raft_consensus.cc:2953] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } unsafe_config_change: true }
I20250815 07:57:37.025422 2101 raft_consensus.cc:2953] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } unsafe_config_change: true }
I20250815 07:57:37.034292 2241 catalog_manager.cc:5582] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 reported cstate change: config changed from index -1 to 3, VOTER fcfe8eecaf124ec9bfe962a883a5459b (127.1.121.194) evicted, no longer has a pending config: VOTER 688aefe7bb8f4cf393dd157b5113d217 (127.1.121.196), VOTER 87807a99b0e24018b7083be5306e7797 (127.1.121.195). New cstate: current_term: 4 leader_uuid: "87807a99b0e24018b7083be5306e7797" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250815 07:57:37.042330 2241 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet ae238bcff75a4bc9952d37cc186c5b3c on TS fcfe8eecaf124ec9bfe962a883a5459b: Not found: failed to reset TS proxy: Could not find TS for UUID fcfe8eecaf124ec9bfe962a883a5459b
I20250815 07:57:37.060979 1967 consensus_queue.cc:237] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: true } } unsafe_config_change: true
I20250815 07:57:37.065698 2101 raft_consensus.cc:1273] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 4 FOLLOWER]: Refusing update from remote peer 87807a99b0e24018b7083be5306e7797: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250815 07:57:37.066933 2340 consensus_queue.cc:1035] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250815 07:57:37.071796 2343 raft_consensus.cc:2953] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER bf8333e9ff054a62936a869121c2eda2 (127.1.121.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: true } } unsafe_config_change: true }
I20250815 07:57:37.073184 2101 raft_consensus.cc:2953] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER bf8333e9ff054a62936a869121c2eda2 (127.1.121.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: true } } unsafe_config_change: true }
I20250815 07:57:37.079530 2226 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae238bcff75a4bc9952d37cc186c5b3c with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250815 07:57:37.081069 1901 consensus_peers.cc:489] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 -> Peer bf8333e9ff054a62936a869121c2eda2 (127.1.121.193:39201): Couldn't send request to peer bf8333e9ff054a62936a869121c2eda2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae238bcff75a4bc9952d37cc186c5b3c. This is attempt 1: this message will repeat every 5th retry.
I20250815 07:57:37.081684 2241 catalog_manager.cc:5582] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 reported cstate change: config changed from index 3 to 5, NON_VOTER bf8333e9ff054a62936a869121c2eda2 (127.1.121.193) added. New cstate: current_term: 4 leader_uuid: "87807a99b0e24018b7083be5306e7797" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250815 07:57:37.095422 2226 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet ae238bcff75a4bc9952d37cc186c5b3c on TS fcfe8eecaf124ec9bfe962a883a5459b failed: Not found: failed to reset TS proxy: Could not find TS for UUID fcfe8eecaf124ec9bfe962a883a5459b
I20250815 07:57:37.582811 2353 ts_tablet_manager.cc:927] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: Initiating tablet copy from peer 87807a99b0e24018b7083be5306e7797 (127.1.121.195:38903)
I20250815 07:57:37.584708 2353 tablet_copy_client.cc:323] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: tablet copy: Beginning tablet copy session from remote peer at address 127.1.121.195:38903
I20250815 07:57:37.593261 1987 tablet_copy_service.cc:140] P 87807a99b0e24018b7083be5306e7797: Received BeginTabletCopySession request for tablet ae238bcff75a4bc9952d37cc186c5b3c from peer bf8333e9ff054a62936a869121c2eda2 ({username='slave'} at 127.1.121.193:50997)
I20250815 07:57:37.593634 1987 tablet_copy_service.cc:161] P 87807a99b0e24018b7083be5306e7797: Beginning new tablet copy session on tablet ae238bcff75a4bc9952d37cc186c5b3c from peer bf8333e9ff054a62936a869121c2eda2 at {username='slave'} at 127.1.121.193:50997: session id = bf8333e9ff054a62936a869121c2eda2-ae238bcff75a4bc9952d37cc186c5b3c
I20250815 07:57:37.597899 1987 tablet_copy_source_session.cc:215] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 07:57:37.602118 2353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae238bcff75a4bc9952d37cc186c5b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:37.617655 2353 tablet_copy_client.cc:806] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: tablet copy: Starting download of 0 data blocks...
I20250815 07:57:37.618093 2353 tablet_copy_client.cc:670] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: tablet copy: Starting download of 1 WAL segments...
I20250815 07:57:37.620937 2353 tablet_copy_client.cc:538] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 07:57:37.625324 2353 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: Bootstrap starting.
I20250815 07:57:37.635037 2353 log.cc:826] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:37.644234 2353 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:57:37.644767 2353 tablet_bootstrap.cc:492] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: Bootstrap complete.
I20250815 07:57:37.645182 2353 ts_tablet_manager.cc:1397] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.005s
I20250815 07:57:37.659646 2353 raft_consensus.cc:357] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: true } } unsafe_config_change: true
I20250815 07:57:37.660357 2353 raft_consensus.cc:738] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: bf8333e9ff054a62936a869121c2eda2, State: Initialized, Role: LEARNER
I20250815 07:57:37.660902 2353 consensus_queue.cc:260] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: true } } unsafe_config_change: true
I20250815 07:57:37.663697 2353 ts_tablet_manager.cc:1428] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
I20250815 07:57:37.665033 1987 tablet_copy_service.cc:342] P 87807a99b0e24018b7083be5306e7797: Request end of tablet copy session bf8333e9ff054a62936a869121c2eda2-ae238bcff75a4bc9952d37cc186c5b3c received from {username='slave'} at 127.1.121.193:50997
I20250815 07:57:37.665338 1987 tablet_copy_service.cc:434] P 87807a99b0e24018b7083be5306e7797: ending tablet copy session bf8333e9ff054a62936a869121c2eda2-ae238bcff75a4bc9952d37cc186c5b3c on tablet ae238bcff75a4bc9952d37cc186c5b3c with peer bf8333e9ff054a62936a869121c2eda2
I20250815 07:57:37.988924 1701 raft_consensus.cc:1215] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250815 07:57:38.261480 2226 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet ae238bcff75a4bc9952d37cc186c5b3c on TS fcfe8eecaf124ec9bfe962a883a5459b failed: Not found: failed to reset TS proxy: Could not find TS for UUID fcfe8eecaf124ec9bfe962a883a5459b
I20250815 07:57:38.539844 2362 raft_consensus.cc:1062] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797: attempting to promote NON_VOTER bf8333e9ff054a62936a869121c2eda2 to VOTER
I20250815 07:57:38.541553 2362 consensus_queue.cc:237] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: false } } unsafe_config_change: true
I20250815 07:57:38.545629 1701 raft_consensus.cc:1273] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2 [term 4 LEARNER]: Refusing update from remote peer 87807a99b0e24018b7083be5306e7797: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250815 07:57:38.546317 2101 raft_consensus.cc:1273] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 4 FOLLOWER]: Refusing update from remote peer 87807a99b0e24018b7083be5306e7797: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250815 07:57:38.547015 2363 consensus_queue.cc:1035] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250815 07:57:38.547824 2365 consensus_queue.cc:1035] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [LEADER]: Connected to new peer: Peer: permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250815 07:57:38.553966 2363 raft_consensus.cc:2953] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, bf8333e9ff054a62936a869121c2eda2 (127.1.121.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: false } } unsafe_config_change: true }
I20250815 07:57:38.555508 1701 raft_consensus.cc:2953] T ae238bcff75a4bc9952d37cc186c5b3c P bf8333e9ff054a62936a869121c2eda2 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, bf8333e9ff054a62936a869121c2eda2 (127.1.121.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: false } } unsafe_config_change: true }
I20250815 07:57:38.556344 2101 raft_consensus.cc:2953] T ae238bcff75a4bc9952d37cc186c5b3c P 688aefe7bb8f4cf393dd157b5113d217 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, bf8333e9ff054a62936a869121c2eda2 (127.1.121.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: false } } unsafe_config_change: true }
I20250815 07:57:38.565152 2241 catalog_manager.cc:5582] T ae238bcff75a4bc9952d37cc186c5b3c P 87807a99b0e24018b7083be5306e7797 reported cstate change: config changed from index 5 to 6, bf8333e9ff054a62936a869121c2eda2 (127.1.121.193) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "87807a99b0e24018b7083be5306e7797" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87807a99b0e24018b7083be5306e7797" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 38903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf8333e9ff054a62936a869121c2eda2" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 39201 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250815 07:57:38.575024 1511 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250815 07:57:38.589713 1511 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "ae238bcff75a4bc9952d37cc186c5b3c"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "688aefe7bb8f4cf393dd157b5113d217"
rpc_addresses {
host: "127.1.121.196"
port: 46435
}
}
ts_infos {
permanent_uuid: "87807a99b0e24018b7083be5306e7797"
rpc_addresses {
host: "127.1.121.195"
port: 38903
}
}
ts_infos {
permanent_uuid: "bf8333e9ff054a62936a869121c2eda2"
rpc_addresses {
host: "127.1.121.193"
port: 39201
}
}
I20250815 07:57:38.591827 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 1617
I20250815 07:57:38.613768 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 1883
I20250815 07:57:38.636487 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 2016
I20250815 07:57:38.658110 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 2185
2025-08-15T07:57:38Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19533 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250815 07:57:38.709272 1511 test_util.cc:276] Using random seed: -1100983550
I20250815 07:57:38.714316 1511 ts_itest-base.cc:115] Starting cluster with:
I20250815 07:57:38.714458 1511 ts_itest-base.cc:116] --------------
I20250815 07:57:38.714612 1511 ts_itest-base.cc:117] 3 tablet servers
I20250815 07:57:38.714740 1511 ts_itest-base.cc:118] 3 replicas per TS
I20250815 07:57:38.714871 1511 ts_itest-base.cc:119] --------------
2025-08-15T07:57:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T07:57:38Z Disabled control of system clock
I20250815 07:57:38.744843 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:34351
--webserver_interface=127.1.121.254
--webserver_port=0
--builtin_ntp_servers=127.1.121.212:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:34351
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250815 07:57:39.001335 2383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:39.001860 2383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:39.002254 2383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:39.028846 2383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:57:39.029070 2383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:39.029248 2383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:57:39.029423 2383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:57:39.058380 2383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:34351
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:34351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:39.059468 2383 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:39.060914 2383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:39.070176 2389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:39.070340 2390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:40.130973 2392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:40.133342 2391 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1058 milliseconds
W20250815 07:57:40.133934 2383 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.064s user 0.350s sys 0.706s
W20250815 07:57:40.134186 2383 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.064s user 0.350s sys 0.706s
I20250815 07:57:40.134392 2383 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:40.135353 2383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:40.137822 2383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:40.139174 2383 hybrid_clock.cc:648] HybridClock initialized: now 1755244660139143 us; error 29 us; skew 500 ppm
I20250815 07:57:40.139954 2383 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:40.147135 2383 webserver.cc:480] Webserver started at http://127.1.121.254:39387/ using document root <none> and password file <none>
I20250815 07:57:40.148063 2383 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:40.148272 2383 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:40.148694 2383 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:40.152611 2383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f4dd15ba99924900a6e41f9d3df70e87"
format_stamp: "Formatted at 2025-08-15 07:57:40 on dist-test-slave-dlgj"
I20250815 07:57:40.153697 2383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f4dd15ba99924900a6e41f9d3df70e87"
format_stamp: "Formatted at 2025-08-15 07:57:40 on dist-test-slave-dlgj"
I20250815 07:57:40.161024 2383 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250815 07:57:40.166462 2400 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:40.167559 2383 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250815 07:57:40.167936 2383 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f4dd15ba99924900a6e41f9d3df70e87"
format_stamp: "Formatted at 2025-08-15 07:57:40 on dist-test-slave-dlgj"
I20250815 07:57:40.168318 2383 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:40.233407 2383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:40.234622 2383 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:40.234987 2383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:40.294060 2383 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:34351
I20250815 07:57:40.294122 2451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:34351 every 8 connection(s)
I20250815 07:57:40.296447 2383 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:57:40.300753 2452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:40.306418 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 2383
I20250815 07:57:40.307070 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 07:57:40.318111 2452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87: Bootstrap starting.
I20250815 07:57:40.323709 2452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:40.325095 2452 log.cc:826] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:40.328735 2452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87: No bootstrap required, opened a new log
I20250815 07:57:40.344029 2452 raft_consensus.cc:357] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4dd15ba99924900a6e41f9d3df70e87" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 34351 } }
I20250815 07:57:40.344563 2452 raft_consensus.cc:383] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:40.344755 2452 raft_consensus.cc:738] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4dd15ba99924900a6e41f9d3df70e87, State: Initialized, Role: FOLLOWER
I20250815 07:57:40.345295 2452 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4dd15ba99924900a6e41f9d3df70e87" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 34351 } }
I20250815 07:57:40.345721 2452 raft_consensus.cc:397] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:57:40.345932 2452 raft_consensus.cc:491] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:57:40.346204 2452 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:40.349860 2452 raft_consensus.cc:513] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4dd15ba99924900a6e41f9d3df70e87" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 34351 } }
I20250815 07:57:40.350397 2452 leader_election.cc:304] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4dd15ba99924900a6e41f9d3df70e87; no voters:
I20250815 07:57:40.351755 2452 leader_election.cc:290] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:57:40.352398 2457 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:57:40.354236 2457 raft_consensus.cc:695] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [term 1 LEADER]: Becoming Leader. State: Replica: f4dd15ba99924900a6e41f9d3df70e87, State: Running, Role: LEADER
I20250815 07:57:40.354856 2457 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4dd15ba99924900a6e41f9d3df70e87" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 34351 } }
I20250815 07:57:40.355643 2452 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:57:40.361377 2458 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4dd15ba99924900a6e41f9d3df70e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4dd15ba99924900a6e41f9d3df70e87" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 34351 } } }
I20250815 07:57:40.362138 2458 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [sys.catalog]: This master's current role is: LEADER
I20250815 07:57:40.363561 2459 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4dd15ba99924900a6e41f9d3df70e87. Latest consensus state: current_term: 1 leader_uuid: "f4dd15ba99924900a6e41f9d3df70e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4dd15ba99924900a6e41f9d3df70e87" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 34351 } } }
I20250815 07:57:40.364255 2459 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87 [sys.catalog]: This master's current role is: LEADER
I20250815 07:57:40.366276 2465 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:57:40.376957 2465 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:57:40.391263 2465 catalog_manager.cc:1349] Generated new cluster ID: 17692d7368cf490487558be8c5991759
I20250815 07:57:40.391518 2465 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:57:40.430369 2465 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 07:57:40.431502 2465 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:57:40.442548 2465 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f4dd15ba99924900a6e41f9d3df70e87: Generated new TSK 0
I20250815 07:57:40.443230 2465 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:57:40.456729 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:0
--local_ip_for_outbound_sockets=127.1.121.193
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:34351
--builtin_ntp_servers=127.1.121.212:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250815 07:57:40.719861 2476 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 07:57:40.720471 2476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:40.720707 2476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:40.721131 2476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:40.748160 2476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:40.748860 2476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:57:40.778373 2476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:34351
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:40.779572 2476 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:40.781037 2476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:40.792059 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:40.793094 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:42.065186 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:42.066133 2484 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1272 milliseconds
W20250815 07:57:42.067773 2476 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.275s user 0.414s sys 0.844s
W20250815 07:57:42.068140 2476 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.275s user 0.414s sys 0.847s
I20250815 07:57:42.068392 2476 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:42.069726 2476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:42.072526 2476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:42.074002 2476 hybrid_clock.cc:648] HybridClock initialized: now 1755244662073926 us; error 70 us; skew 500 ppm
I20250815 07:57:42.075018 2476 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:42.084329 2476 webserver.cc:480] Webserver started at http://127.1.121.193:38079/ using document root <none> and password file <none>
I20250815 07:57:42.085726 2476 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:42.086001 2476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:42.086613 2476 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:42.094525 2476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "35ffec2c3ba548548be542b2be472731"
format_stamp: "Formatted at 2025-08-15 07:57:42 on dist-test-slave-dlgj"
I20250815 07:57:42.095916 2476 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "35ffec2c3ba548548be542b2be472731"
format_stamp: "Formatted at 2025-08-15 07:57:42 on dist-test-slave-dlgj"
I20250815 07:57:42.105589 2476 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20250815 07:57:42.112890 2492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:42.114221 2476 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250815 07:57:42.114611 2476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "35ffec2c3ba548548be542b2be472731"
format_stamp: "Formatted at 2025-08-15 07:57:42 on dist-test-slave-dlgj"
I20250815 07:57:42.115068 2476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:42.196969 2476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:42.198964 2476 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:42.199501 2476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:42.202908 2476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:42.207068 2476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:42.207278 2476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:42.207511 2476 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:42.207655 2476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:42.355347 2476 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:37867
I20250815 07:57:42.355415 2604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:37867 every 8 connection(s)
I20250815 07:57:42.358704 2476 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:57:42.365676 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 2476
I20250815 07:57:42.366117 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 07:57:42.372807 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:0
--local_ip_for_outbound_sockets=127.1.121.194
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:34351
--builtin_ntp_servers=127.1.121.212:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250815 07:57:42.383718 2605 heartbeater.cc:344] Connected to a master server at 127.1.121.254:34351
I20250815 07:57:42.384114 2605 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:42.385035 2605 heartbeater.cc:507] Master 127.1.121.254:34351 requested a full tablet report, sending...
I20250815 07:57:42.387285 2417 ts_manager.cc:194] Registered new tserver with Master: 35ffec2c3ba548548be542b2be472731 (127.1.121.193:37867)
I20250815 07:57:42.389087 2417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:33313
W20250815 07:57:42.664005 2609 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 07:57:42.664599 2609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:42.664778 2609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:42.665133 2609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:42.692412 2609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:42.693085 2609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:57:42.724403 2609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:34351
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:42.725490 2609 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:42.726903 2609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:42.737398 2615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:43.392091 2605 heartbeater.cc:499] Master 127.1.121.254:34351 was elected leader, sending a full tablet report...
W20250815 07:57:44.141031 2614 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 2609
W20250815 07:57:44.292030 2614 kernel_stack_watchdog.cc:198] Thread 2609 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 07:57:42.737901 2616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:44.296659 2609 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.559s user 0.581s sys 0.977s
W20250815 07:57:44.297020 2609 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.559s user 0.582s sys 0.977s
W20250815 07:57:44.298439 2617 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1560 milliseconds
W20250815 07:57:44.298889 2618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:44.299247 2609 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:44.302958 2609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:44.304822 2609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:44.306119 2609 hybrid_clock.cc:648] HybridClock initialized: now 1755244664306081 us; error 35 us; skew 500 ppm
I20250815 07:57:44.306823 2609 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:44.312026 2609 webserver.cc:480] Webserver started at http://127.1.121.194:44023/ using document root <none> and password file <none>
I20250815 07:57:44.312860 2609 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:44.313057 2609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:44.313438 2609 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:44.317334 2609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf"
format_stamp: "Formatted at 2025-08-15 07:57:44 on dist-test-slave-dlgj"
I20250815 07:57:44.318295 2609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf"
format_stamp: "Formatted at 2025-08-15 07:57:44 on dist-test-slave-dlgj"
I20250815 07:57:44.324352 2609 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250815 07:57:44.329203 2625 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:44.330029 2609 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250815 07:57:44.330291 2609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf"
format_stamp: "Formatted at 2025-08-15 07:57:44 on dist-test-slave-dlgj"
I20250815 07:57:44.330560 2609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:44.374620 2609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:44.375836 2609 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:44.376222 2609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:44.378433 2609 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:44.382110 2609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:44.382297 2609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:44.382498 2609 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:44.382632 2609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:44.501664 2609 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:41649
I20250815 07:57:44.501739 2737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:41649 every 8 connection(s)
I20250815 07:57:44.503861 2609 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 07:57:44.512609 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 2609
I20250815 07:57:44.512920 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 07:57:44.518340 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:0
--local_ip_for_outbound_sockets=127.1.121.195
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:34351
--builtin_ntp_servers=127.1.121.212:32873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250815 07:57:44.522570 2738 heartbeater.cc:344] Connected to a master server at 127.1.121.254:34351
I20250815 07:57:44.522934 2738 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:44.523846 2738 heartbeater.cc:507] Master 127.1.121.254:34351 requested a full tablet report, sending...
I20250815 07:57:44.525815 2417 ts_manager.cc:194] Registered new tserver with Master: 3c2ba0e4d30d41c49a1d3b58df9741cf (127.1.121.194:41649)
I20250815 07:57:44.527525 2417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:55329
W20250815 07:57:44.785334 2742 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250815 07:57:44.785817 2742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:44.786055 2742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:44.786440 2742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:44.812779 2742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:44.813421 2742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:57:44.842446 2742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:32873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:34351
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:44.843478 2742 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:44.844823 2742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:44.854322 2748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:45.530184 2738 heartbeater.cc:499] Master 127.1.121.254:34351 was elected leader, sending a full tablet report...
W20250815 07:57:44.856070 2749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:46.097445 2750 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1238 milliseconds
W20250815 07:57:46.098932 2751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:46.099217 2742 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.244s user 0.490s sys 0.744s
W20250815 07:57:46.099573 2742 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.244s user 0.490s sys 0.744s
I20250815 07:57:46.099794 2742 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:46.100800 2742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:46.102886 2742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:46.104214 2742 hybrid_clock.cc:648] HybridClock initialized: now 1755244666104166 us; error 41 us; skew 500 ppm
I20250815 07:57:46.104885 2742 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:46.115432 2742 webserver.cc:480] Webserver started at http://127.1.121.195:34777/ using document root <none> and password file <none>
I20250815 07:57:46.116289 2742 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:46.116487 2742 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:46.116909 2742 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:46.120972 2742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b5b3d5e66eab4b2fb907bcdd256500a9"
format_stamp: "Formatted at 2025-08-15 07:57:46 on dist-test-slave-dlgj"
I20250815 07:57:46.121933 2742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b5b3d5e66eab4b2fb907bcdd256500a9"
format_stamp: "Formatted at 2025-08-15 07:57:46 on dist-test-slave-dlgj"
I20250815 07:57:46.128700 2742 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250815 07:57:46.134001 2758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:46.134986 2742 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250815 07:57:46.135273 2742 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b5b3d5e66eab4b2fb907bcdd256500a9"
format_stamp: "Formatted at 2025-08-15 07:57:46 on dist-test-slave-dlgj"
I20250815 07:57:46.135558 2742 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:46.204680 2742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:46.206008 2742 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:46.206427 2742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:46.209228 2742 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:46.213021 2742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:46.213223 2742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:46.213433 2742 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:46.213575 2742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:46.335382 2742 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:33153
I20250815 07:57:46.335462 2870 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:33153 every 8 connection(s)
I20250815 07:57:46.337728 2742 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 07:57:46.348040 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 2742
I20250815 07:57:46.348492 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 07:57:46.356083 2871 heartbeater.cc:344] Connected to a master server at 127.1.121.254:34351
I20250815 07:57:46.356495 2871 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:46.357424 2871 heartbeater.cc:507] Master 127.1.121.254:34351 requested a full tablet report, sending...
I20250815 07:57:46.359210 2417 ts_manager.cc:194] Registered new tserver with Master: b5b3d5e66eab4b2fb907bcdd256500a9 (127.1.121.195:33153)
I20250815 07:57:46.360316 2417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:49139
I20250815 07:57:46.366626 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:57:46.395766 2417 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44812:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 07:57:46.413007 2417 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 07:57:46.451483 2540 tablet_service.cc:1468] Processing CreateTablet for tablet bd5948d5155a4f069329dfbe9678b2be (DEFAULT_TABLE table=TestTable [id=533305bf6c964e11937eeb58f0402a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:46.453111 2540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5948d5155a4f069329dfbe9678b2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:46.455508 2673 tablet_service.cc:1468] Processing CreateTablet for tablet bd5948d5155a4f069329dfbe9678b2be (DEFAULT_TABLE table=TestTable [id=533305bf6c964e11937eeb58f0402a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:46.457415 2673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5948d5155a4f069329dfbe9678b2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:46.456919 2806 tablet_service.cc:1468] Processing CreateTablet for tablet bd5948d5155a4f069329dfbe9678b2be (DEFAULT_TABLE table=TestTable [id=533305bf6c964e11937eeb58f0402a71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:46.458516 2806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5948d5155a4f069329dfbe9678b2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:46.475145 2890 tablet_bootstrap.cc:492] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf: Bootstrap starting.
I20250815 07:57:46.476768 2891 tablet_bootstrap.cc:492] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731: Bootstrap starting.
I20250815 07:57:46.482301 2890 tablet_bootstrap.cc:654] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:46.483276 2891 tablet_bootstrap.cc:654] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:46.484391 2890 log.cc:826] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:46.484848 2892 tablet_bootstrap.cc:492] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9: Bootstrap starting.
I20250815 07:57:46.485330 2891 log.cc:826] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:46.489351 2890 tablet_bootstrap.cc:492] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf: No bootstrap required, opened a new log
I20250815 07:57:46.489760 2890 ts_tablet_manager.cc:1397] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250815 07:57:46.490620 2892 tablet_bootstrap.cc:654] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:46.491012 2891 tablet_bootstrap.cc:492] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731: No bootstrap required, opened a new log
I20250815 07:57:46.491426 2891 ts_tablet_manager.cc:1397] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20250815 07:57:46.493602 2892 log.cc:826] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:46.497519 2892 tablet_bootstrap.cc:492] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9: No bootstrap required, opened a new log
I20250815 07:57:46.497840 2892 ts_tablet_manager.cc:1397] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20250815 07:57:46.514042 2891 raft_consensus.cc:357] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:46.514042 2890 raft_consensus.cc:357] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:46.514845 2890 raft_consensus.cc:738] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2ba0e4d30d41c49a1d3b58df9741cf, State: Initialized, Role: FOLLOWER
I20250815 07:57:46.515012 2891 raft_consensus.cc:738] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ffec2c3ba548548be542b2be472731, State: Initialized, Role: FOLLOWER
I20250815 07:57:46.515530 2890 consensus_queue.cc:260] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:46.515679 2891 consensus_queue.cc:260] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:46.522727 2891 ts_tablet_manager.cc:1428] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731: Time spent starting tablet: real 0.031s user 0.025s sys 0.003s
I20250815 07:57:46.525305 2892 raft_consensus.cc:357] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:46.526132 2892 raft_consensus.cc:738] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b3d5e66eab4b2fb907bcdd256500a9, State: Initialized, Role: FOLLOWER
I20250815 07:57:46.526854 2892 consensus_queue.cc:260] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:46.528967 2890 ts_tablet_manager.cc:1428] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf: Time spent starting tablet: real 0.039s user 0.025s sys 0.007s
I20250815 07:57:46.530325 2871 heartbeater.cc:499] Master 127.1.121.254:34351 was elected leader, sending a full tablet report...
I20250815 07:57:46.531504 2892 ts_tablet_manager.cc:1428] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9: Time spent starting tablet: real 0.033s user 0.029s sys 0.000s
I20250815 07:57:46.553942 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:57:46.556737 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 35ffec2c3ba548548be542b2be472731 to finish bootstrapping
I20250815 07:57:46.567626 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3c2ba0e4d30d41c49a1d3b58df9741cf to finish bootstrapping
I20250815 07:57:46.576205 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b5b3d5e66eab4b2fb907bcdd256500a9 to finish bootstrapping
W20250815 07:57:46.592979 2872 tablet.cc:2378] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:57:46.608939 2560 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bd5948d5155a4f069329dfbe9678b2be"
dest_uuid: "35ffec2c3ba548548be542b2be472731"
from {username='slave'} at 127.0.0.1:43264
I20250815 07:57:46.609418 2560 raft_consensus.cc:491] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250815 07:57:46.609661 2560 raft_consensus.cc:3058] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:46.613703 2560 raft_consensus.cc:513] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:46.615664 2560 leader_election.cc:290] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [CANDIDATE]: Term 1 election: Requested vote from peers b5b3d5e66eab4b2fb907bcdd256500a9 (127.1.121.195:33153), 3c2ba0e4d30d41c49a1d3b58df9741cf (127.1.121.194:41649)
W20250815 07:57:46.617303 2606 tablet.cc:2378] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:57:46.623206 1511 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250815 07:57:46.625767 2826 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5948d5155a4f069329dfbe9678b2be" candidate_uuid: "35ffec2c3ba548548be542b2be472731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9"
I20250815 07:57:46.626401 2826 raft_consensus.cc:3058] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:46.628154 2693 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5948d5155a4f069329dfbe9678b2be" candidate_uuid: "35ffec2c3ba548548be542b2be472731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf"
I20250815 07:57:46.628616 2693 raft_consensus.cc:3058] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:46.630416 2826 raft_consensus.cc:2466] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ffec2c3ba548548be542b2be472731 in term 1.
I20250815 07:57:46.631239 2494 leader_election.cc:304] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ffec2c3ba548548be542b2be472731, b5b3d5e66eab4b2fb907bcdd256500a9; no voters:
I20250815 07:57:46.631872 2897 raft_consensus.cc:2802] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:57:46.632429 2693 raft_consensus.cc:2466] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ffec2c3ba548548be542b2be472731 in term 1.
I20250815 07:57:46.633430 2897 raft_consensus.cc:695] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 1 LEADER]: Becoming Leader. State: Replica: 35ffec2c3ba548548be542b2be472731, State: Running, Role: LEADER
I20250815 07:57:46.634048 2897 consensus_queue.cc:237] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:46.642026 2417 catalog_manager.cc:5582] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ffec2c3ba548548be542b2be472731 (127.1.121.193). New cstate: current_term: 1 leader_uuid: "35ffec2c3ba548548be542b2be472731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } health_report { overall_health: UNKNOWN } } }
I20250815 07:57:46.728474 1511 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250815 07:57:46.759815 2739 tablet.cc:2378] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:57:46.933295 1511 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250815 07:57:47.072979 2897 consensus_queue.cc:1035] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 07:57:47.085641 2897 consensus_queue.cc:1035] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 07:57:48.687304 2560 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bd5948d5155a4f069329dfbe9678b2be"
dest_uuid: "35ffec2c3ba548548be542b2be472731"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43274
I20250815 07:57:48.687733 2560 raft_consensus.cc:604] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 1 LEADER]: Received request to transfer leadership
I20250815 07:57:48.959057 2932 raft_consensus.cc:991] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731: : Instructing follower b5b3d5e66eab4b2fb907bcdd256500a9 to start an election
I20250815 07:57:48.959422 2897 raft_consensus.cc:1079] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 1 LEADER]: Signalling peer b5b3d5e66eab4b2fb907bcdd256500a9 to start an election
I20250815 07:57:48.960580 2826 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bd5948d5155a4f069329dfbe9678b2be"
dest_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9"
from {username='slave'} at 127.1.121.193:33717
I20250815 07:57:48.960999 2826 raft_consensus.cc:491] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250815 07:57:48.961205 2826 raft_consensus.cc:3058] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 1 FOLLOWER]: Advancing to term 2
I20250815 07:57:48.965173 2826 raft_consensus.cc:513] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:48.966820 2826 leader_election.cc:290] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [CANDIDATE]: Term 2 election: Requested vote from peers 35ffec2c3ba548548be542b2be472731 (127.1.121.193:37867), 3c2ba0e4d30d41c49a1d3b58df9741cf (127.1.121.194:41649)
I20250815 07:57:48.975576 2560 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5948d5155a4f069329dfbe9678b2be" candidate_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35ffec2c3ba548548be542b2be472731"
I20250815 07:57:48.975975 2560 raft_consensus.cc:3053] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 1 LEADER]: Stepping down as leader of term 1
I20250815 07:57:48.976218 2560 raft_consensus.cc:738] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35ffec2c3ba548548be542b2be472731, State: Running, Role: LEADER
I20250815 07:57:48.976670 2560 consensus_queue.cc:260] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:48.977542 2560 raft_consensus.cc:3058] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 1 FOLLOWER]: Advancing to term 2
I20250815 07:57:48.978055 2693 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5948d5155a4f069329dfbe9678b2be" candidate_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf"
I20250815 07:57:48.978436 2693 raft_consensus.cc:3058] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf [term 1 FOLLOWER]: Advancing to term 2
I20250815 07:57:48.981191 2560 raft_consensus.cc:2466] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b3d5e66eab4b2fb907bcdd256500a9 in term 2.
I20250815 07:57:48.981931 2760 leader_election.cc:304] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ffec2c3ba548548be542b2be472731, b5b3d5e66eab4b2fb907bcdd256500a9; no voters:
I20250815 07:57:48.982059 2693 raft_consensus.cc:2466] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b3d5e66eab4b2fb907bcdd256500a9 in term 2.
I20250815 07:57:48.983812 2936 raft_consensus.cc:2802] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 07:57:48.985085 2936 raft_consensus.cc:695] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [term 2 LEADER]: Becoming Leader. State: Replica: b5b3d5e66eab4b2fb907bcdd256500a9, State: Running, Role: LEADER
I20250815 07:57:48.985936 2936 consensus_queue.cc:237] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } }
I20250815 07:57:48.999236 2415 catalog_manager.cc:5582] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 reported cstate change: term changed from 1 to 2, leader changed from 35ffec2c3ba548548be542b2be472731 (127.1.121.193) to b5b3d5e66eab4b2fb907bcdd256500a9 (127.1.121.195). New cstate: current_term: 2 leader_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5b3d5e66eab4b2fb907bcdd256500a9" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 33153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 } health_report { overall_health: UNKNOWN } } }
I20250815 07:57:49.466142 2560 raft_consensus.cc:1273] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 2 FOLLOWER]: Refusing update from remote peer b5b3d5e66eab4b2fb907bcdd256500a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 07:57:49.467196 2936 consensus_queue.cc:1035] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ffec2c3ba548548be542b2be472731" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250815 07:57:49.479216 2693 raft_consensus.cc:1273] T bd5948d5155a4f069329dfbe9678b2be P 3c2ba0e4d30d41c49a1d3b58df9741cf [term 2 FOLLOWER]: Refusing update from remote peer b5b3d5e66eab4b2fb907bcdd256500a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250815 07:57:49.480233 2936 consensus_queue.cc:1035] T bd5948d5155a4f069329dfbe9678b2be P b5b3d5e66eab4b2fb907bcdd256500a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c2ba0e4d30d41c49a1d3b58df9741cf" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250815 07:57:51.170857 2560 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bd5948d5155a4f069329dfbe9678b2be"
dest_uuid: "35ffec2c3ba548548be542b2be472731"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43276
I20250815 07:57:51.171713 2560 raft_consensus.cc:604] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 2 FOLLOWER]: Received request to transfer leadership
I20250815 07:57:51.172160 2560 raft_consensus.cc:612] T bd5948d5155a4f069329dfbe9678b2be P 35ffec2c3ba548548be542b2be472731 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250815 07:57:52.204691 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 2476
I20250815 07:57:52.226132 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 2609
I20250815 07:57:52.249258 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 2742
I20250815 07:57:52.273289 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 2383
2025-08-15T07:57:52Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13614 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250815 07:57:52.323805 1511 test_util.cc:276] Using random seed: -1087369020
I20250815 07:57:52.327418 1511 ts_itest-base.cc:115] Starting cluster with:
I20250815 07:57:52.327658 1511 ts_itest-base.cc:116] --------------
I20250815 07:57:52.327832 1511 ts_itest-base.cc:117] 3 tablet servers
I20250815 07:57:52.327970 1511 ts_itest-base.cc:118] 3 replicas per TS
I20250815 07:57:52.328172 1511 ts_itest-base.cc:119] --------------
2025-08-15T07:57:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T07:57:52Z Disabled control of system clock
I20250815 07:57:52.357537 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46523
--webserver_interface=127.1.121.254
--webserver_port=0
--builtin_ntp_servers=127.1.121.212:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:46523 with env {}
W20250815 07:57:52.611789 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:52.612316 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:52.612732 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:52.639079 2980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:57:52.639355 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:52.639575 2980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:57:52.639780 2980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:57:52.669443 2980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:46523
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:52.670557 2980 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:52.671933 2980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:52.681120 2986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:52.681579 2987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:53.733552 2989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:53.735344 2988 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1050 milliseconds
I20250815 07:57:53.735419 2980 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:53.736541 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:53.739260 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:53.740628 2980 hybrid_clock.cc:648] HybridClock initialized: now 1755244673740589 us; error 48 us; skew 500 ppm
I20250815 07:57:53.741315 2980 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:53.746841 2980 webserver.cc:480] Webserver started at http://127.1.121.254:40039/ using document root <none> and password file <none>
I20250815 07:57:53.747604 2980 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:53.747777 2980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:53.748195 2980 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:53.752828 2980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9656c4ce20914c36af5921d9251c3657"
format_stamp: "Formatted at 2025-08-15 07:57:53 on dist-test-slave-dlgj"
I20250815 07:57:53.753760 2980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9656c4ce20914c36af5921d9251c3657"
format_stamp: "Formatted at 2025-08-15 07:57:53 on dist-test-slave-dlgj"
I20250815 07:57:53.760183 2980 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250815 07:57:53.764597 2996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:53.765496 2980 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250815 07:57:53.765770 2980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9656c4ce20914c36af5921d9251c3657"
format_stamp: "Formatted at 2025-08-15 07:57:53 on dist-test-slave-dlgj"
I20250815 07:57:53.766038 2980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:53.815928 2980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:53.817216 2980 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:53.817605 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:53.878386 2980 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:46523
I20250815 07:57:53.878459 3047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:46523 every 8 connection(s)
I20250815 07:57:53.880765 2980 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:57:53.885093 3048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:53.890456 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 2980
I20250815 07:57:53.890772 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 07:57:53.901027 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657: Bootstrap starting.
I20250815 07:57:53.906642 3048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:53.908471 3048 log.cc:826] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:53.912643 3048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657: No bootstrap required, opened a new log
I20250815 07:57:53.927836 3048 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9656c4ce20914c36af5921d9251c3657" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46523 } }
I20250815 07:57:53.928463 3048 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:53.928682 3048 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9656c4ce20914c36af5921d9251c3657, State: Initialized, Role: FOLLOWER
I20250815 07:57:53.929251 3048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9656c4ce20914c36af5921d9251c3657" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46523 } }
I20250815 07:57:53.929680 3048 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:57:53.929894 3048 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:57:53.930153 3048 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:57:53.933501 3048 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9656c4ce20914c36af5921d9251c3657" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46523 } }
I20250815 07:57:53.934054 3048 leader_election.cc:304] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9656c4ce20914c36af5921d9251c3657; no voters:
I20250815 07:57:53.935508 3048 leader_election.cc:290] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:57:53.936180 3053 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:57:53.938175 3053 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [term 1 LEADER]: Becoming Leader. State: Replica: 9656c4ce20914c36af5921d9251c3657, State: Running, Role: LEADER
I20250815 07:57:53.938830 3053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9656c4ce20914c36af5921d9251c3657" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46523 } }
I20250815 07:57:53.939698 3048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:57:53.947397 3055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9656c4ce20914c36af5921d9251c3657. Latest consensus state: current_term: 1 leader_uuid: "9656c4ce20914c36af5921d9251c3657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9656c4ce20914c36af5921d9251c3657" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46523 } } }
I20250815 07:57:53.948036 3055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [sys.catalog]: This master's current role is: LEADER
I20250815 07:57:53.947683 3054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9656c4ce20914c36af5921d9251c3657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9656c4ce20914c36af5921d9251c3657" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46523 } } }
I20250815 07:57:53.948390 3054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657 [sys.catalog]: This master's current role is: LEADER
I20250815 07:57:53.950760 3061 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:57:53.960588 3061 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:57:53.973867 3061 catalog_manager.cc:1349] Generated new cluster ID: 47de796b5bb047e69ec6398364b497a7
I20250815 07:57:53.974051 3061 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:57:53.983063 3061 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 07:57:53.984165 3061 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:57:53.994035 3061 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9656c4ce20914c36af5921d9251c3657: Generated new TSK 0
I20250815 07:57:53.994693 3061 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:57:54.013146 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:0
--local_ip_for_outbound_sockets=127.1.121.193
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46523
--builtin_ntp_servers=127.1.121.212:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 07:57:54.281719 3072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:54.282119 3072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:54.282543 3072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:54.308885 3072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:54.309583 3072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:57:54.339056 3072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:54.340284 3072 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:54.341789 3072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:54.352830 3078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:55.758378 3077 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3072
W20250815 07:57:56.120692 3077 kernel_stack_watchdog.cc:198] Thread 3072 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 07:57:56.120934 3072 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.767s user 0.618s sys 1.055s
W20250815 07:57:54.354210 3079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:56.121366 3072 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.768s user 0.619s sys 1.056s
W20250815 07:57:56.121749 3080 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1767 milliseconds
W20250815 07:57:56.123006 3081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:56.122954 3072 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:56.126686 3072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:56.129226 3072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:56.130694 3072 hybrid_clock.cc:648] HybridClock initialized: now 1755244676130639 us; error 41 us; skew 500 ppm
I20250815 07:57:56.131677 3072 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:56.138720 3072 webserver.cc:480] Webserver started at http://127.1.121.193:44107/ using document root <none> and password file <none>
I20250815 07:57:56.139948 3072 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:56.140249 3072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:56.140834 3072 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:56.146772 3072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7b7ea35b17ae4980a3ec913ceb19c674"
format_stamp: "Formatted at 2025-08-15 07:57:56 on dist-test-slave-dlgj"
I20250815 07:57:56.148341 3072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7b7ea35b17ae4980a3ec913ceb19c674"
format_stamp: "Formatted at 2025-08-15 07:57:56 on dist-test-slave-dlgj"
I20250815 07:57:56.156971 3072 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20250815 07:57:56.162395 3088 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:56.163414 3072 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250815 07:57:56.163686 3072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7b7ea35b17ae4980a3ec913ceb19c674"
format_stamp: "Formatted at 2025-08-15 07:57:56 on dist-test-slave-dlgj"
I20250815 07:57:56.163978 3072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:56.217254 3072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:56.219275 3072 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:56.219774 3072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:56.223088 3072 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:56.228736 3072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:56.228962 3072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:56.229194 3072 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:56.229423 3072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:56.382309 3072 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:42137
I20250815 07:57:56.382395 3200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:42137 every 8 connection(s)
I20250815 07:57:56.384603 3072 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:57:56.393874 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 3072
I20250815 07:57:56.394280 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 07:57:56.400867 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:0
--local_ip_for_outbound_sockets=127.1.121.194
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46523
--builtin_ntp_servers=127.1.121.212:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:57:56.404145 3201 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46523
I20250815 07:57:56.404505 3201 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:56.405326 3201 heartbeater.cc:507] Master 127.1.121.254:46523 requested a full tablet report, sending...
I20250815 07:57:56.407634 3013 ts_manager.cc:194] Registered new tserver with Master: 7b7ea35b17ae4980a3ec913ceb19c674 (127.1.121.193:42137)
I20250815 07:57:56.409739 3013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:41187
W20250815 07:57:56.679495 3205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:56.679917 3205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:56.680424 3205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:56.707341 3205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:56.708149 3205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:57:56.737462 3205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:56.738615 3205 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:56.740005 3205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:56.750245 3211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:57.412482 3201 heartbeater.cc:499] Master 127.1.121.254:46523 was elected leader, sending a full tablet report...
W20250815 07:57:56.754310 3214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:56.751561 3212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:57.755832 3213 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 07:57:57.755865 3205 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:57.759660 3205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:57.762091 3205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:57.763455 3205 hybrid_clock.cc:648] HybridClock initialized: now 1755244677763420 us; error 52 us; skew 500 ppm
I20250815 07:57:57.764137 3205 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:57.770476 3205 webserver.cc:480] Webserver started at http://127.1.121.194:40067/ using document root <none> and password file <none>
I20250815 07:57:57.771217 3205 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:57.771375 3205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:57.771744 3205 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:57.775403 3205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e2f97c4fba2b4eed9456a44b70dfe22f"
format_stamp: "Formatted at 2025-08-15 07:57:57 on dist-test-slave-dlgj"
I20250815 07:57:57.776329 3205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e2f97c4fba2b4eed9456a44b70dfe22f"
format_stamp: "Formatted at 2025-08-15 07:57:57 on dist-test-slave-dlgj"
I20250815 07:57:57.782218 3205 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250815 07:57:57.786754 3221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:57.787571 3205 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250815 07:57:57.787827 3205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e2f97c4fba2b4eed9456a44b70dfe22f"
format_stamp: "Formatted at 2025-08-15 07:57:57 on dist-test-slave-dlgj"
I20250815 07:57:57.788123 3205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:57.830315 3205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:57.831565 3205 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:57.831894 3205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:57.834465 3205 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:57.838056 3205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:57.838250 3205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:57.838442 3205 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:57.838568 3205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:57.961067 3205 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:40613
I20250815 07:57:57.961182 3333 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:40613 every 8 connection(s)
I20250815 07:57:57.963229 3205 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 07:57:57.963860 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 3205
I20250815 07:57:57.964234 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 07:57:57.970635 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:0
--local_ip_for_outbound_sockets=127.1.121.195
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46523
--builtin_ntp_servers=127.1.121.212:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:57:57.982390 3334 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46523
I20250815 07:57:57.982743 3334 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:57.983584 3334 heartbeater.cc:507] Master 127.1.121.254:46523 requested a full tablet report, sending...
I20250815 07:57:57.985471 3013 ts_manager.cc:194] Registered new tserver with Master: e2f97c4fba2b4eed9456a44b70dfe22f (127.1.121.194:40613)
I20250815 07:57:57.986495 3013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:49653
W20250815 07:57:58.237598 3338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:57:58.238086 3338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:57:58.238600 3338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:57:58.265281 3338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:57:58.266417 3338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:57:58.296661 3338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:57:58.297894 3338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:57:58.299345 3338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:57:58.309466 3344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:57:58.989157 3334 heartbeater.cc:499] Master 127.1.121.254:46523 was elected leader, sending a full tablet report...
W20250815 07:57:58.310200 3345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:59.499910 3338 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.190s user 0.399s sys 0.780s
W20250815 07:57:59.500362 3338 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.190s user 0.399s sys 0.780s
W20250815 07:57:59.501288 3347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:57:59.505259 3346 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1192 milliseconds
I20250815 07:57:59.505290 3338 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:57:59.506724 3338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:57:59.509281 3338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:57:59.510731 3338 hybrid_clock.cc:648] HybridClock initialized: now 1755244679510681 us; error 43 us; skew 500 ppm
I20250815 07:57:59.511718 3338 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:57:59.519799 3338 webserver.cc:480] Webserver started at http://127.1.121.195:33607/ using document root <none> and password file <none>
I20250815 07:57:59.521049 3338 fs_manager.cc:362] Metadata directory not provided
I20250815 07:57:59.521317 3338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:57:59.521888 3338 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:57:59.528568 3338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6ca46ee70eb14f85b481346b9fe0fe91"
format_stamp: "Formatted at 2025-08-15 07:57:59 on dist-test-slave-dlgj"
I20250815 07:57:59.529994 3338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6ca46ee70eb14f85b481346b9fe0fe91"
format_stamp: "Formatted at 2025-08-15 07:57:59 on dist-test-slave-dlgj"
I20250815 07:57:59.538841 3338 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20250815 07:57:59.545643 3355 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:59.546864 3338 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250815 07:57:59.547266 3338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6ca46ee70eb14f85b481346b9fe0fe91"
format_stamp: "Formatted at 2025-08-15 07:57:59 on dist-test-slave-dlgj"
I20250815 07:57:59.547652 3338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:57:59.617054 3338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:57:59.618352 3338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:57:59.618758 3338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:57:59.621518 3338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:57:59.625119 3338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:57:59.625317 3338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250815 07:57:59.625511 3338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:57:59.625624 3338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:57:59.739256 3338 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:43425
I20250815 07:57:59.739353 3467 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:43425 every 8 connection(s)
I20250815 07:57:59.741523 3338 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 07:57:59.747614 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 3338
I20250815 07:57:59.748160 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 07:57:59.759665 3468 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46523
I20250815 07:57:59.759990 3468 heartbeater.cc:461] Registering TS with master...
I20250815 07:57:59.760820 3468 heartbeater.cc:507] Master 127.1.121.254:46523 requested a full tablet report, sending...
I20250815 07:57:59.762581 3013 ts_manager.cc:194] Registered new tserver with Master: 6ca46ee70eb14f85b481346b9fe0fe91 (127.1.121.195:43425)
I20250815 07:57:59.763680 3013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:59417
I20250815 07:57:59.766685 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:57:59.796705 3013 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36906:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 07:57:59.812651 3013 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 07:57:59.855690 3269 tablet_service.cc:1468] Processing CreateTablet for tablet d698c0e1196e4050b582f7ae95da9726 (DEFAULT_TABLE table=TestTable [id=a9f6592b89824cc397fd8dc06edf686b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:59.857451 3269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d698c0e1196e4050b582f7ae95da9726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:59.859701 3136 tablet_service.cc:1468] Processing CreateTablet for tablet d698c0e1196e4050b582f7ae95da9726 (DEFAULT_TABLE table=TestTable [id=a9f6592b89824cc397fd8dc06edf686b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:59.861403 3136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d698c0e1196e4050b582f7ae95da9726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:59.865835 3403 tablet_service.cc:1468] Processing CreateTablet for tablet d698c0e1196e4050b582f7ae95da9726 (DEFAULT_TABLE table=TestTable [id=a9f6592b89824cc397fd8dc06edf686b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:57:59.867257 3403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d698c0e1196e4050b582f7ae95da9726. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:57:59.890286 3488 tablet_bootstrap.cc:492] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674: Bootstrap starting.
I20250815 07:57:59.897214 3487 tablet_bootstrap.cc:492] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f: Bootstrap starting.
I20250815 07:57:59.898295 3488 tablet_bootstrap.cc:654] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:59.899026 3489 tablet_bootstrap.cc:492] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91: Bootstrap starting.
I20250815 07:57:59.900871 3488 log.cc:826] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:59.905097 3489 tablet_bootstrap.cc:654] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:59.905719 3487 tablet_bootstrap.cc:654] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f: Neither blocks nor log segments found. Creating new log.
I20250815 07:57:59.907137 3489 log.cc:826] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:59.908656 3487 log.cc:826] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f: Log is configured to *not* fsync() on all Append() calls
I20250815 07:57:59.914189 3488 tablet_bootstrap.cc:492] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674: No bootstrap required, opened a new log
I20250815 07:57:59.914490 3487 tablet_bootstrap.cc:492] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f: No bootstrap required, opened a new log
I20250815 07:57:59.914816 3488 ts_tablet_manager.cc:1397] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.006s
I20250815 07:57:59.914857 3487 ts_tablet_manager.cc:1397] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20250815 07:57:59.923142 3489 tablet_bootstrap.cc:492] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91: No bootstrap required, opened a new log
I20250815 07:57:59.923619 3489 ts_tablet_manager.cc:1397] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.012s
I20250815 07:57:59.939159 3488 raft_consensus.cc:357] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:57:59.940163 3488 raft_consensus.cc:383] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:59.940495 3488 raft_consensus.cc:738] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ea35b17ae4980a3ec913ceb19c674, State: Initialized, Role: FOLLOWER
I20250815 07:57:59.940033 3487 raft_consensus.cc:357] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:57:59.940948 3487 raft_consensus.cc:383] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:59.941255 3487 raft_consensus.cc:738] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f97c4fba2b4eed9456a44b70dfe22f, State: Initialized, Role: FOLLOWER
I20250815 07:57:59.941615 3488 consensus_queue.cc:260] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:57:59.942091 3487 consensus_queue.cc:260] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:57:59.947636 3489 raft_consensus.cc:357] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:57:59.948457 3489 raft_consensus.cc:383] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:57:59.948753 3489 raft_consensus.cc:738] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca46ee70eb14f85b481346b9fe0fe91, State: Initialized, Role: FOLLOWER
I20250815 07:57:59.949584 3489 consensus_queue.cc:260] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:57:59.952028 3488 ts_tablet_manager.cc:1428] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674: Time spent starting tablet: real 0.037s user 0.029s sys 0.004s
I20250815 07:57:59.954398 3487 ts_tablet_manager.cc:1428] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f: Time spent starting tablet: real 0.039s user 0.019s sys 0.018s
I20250815 07:57:59.956923 3468 heartbeater.cc:499] Master 127.1.121.254:46523 was elected leader, sending a full tablet report...
I20250815 07:57:59.958158 3489 ts_tablet_manager.cc:1428] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91: Time spent starting tablet: real 0.034s user 0.016s sys 0.014s
W20250815 07:57:59.969254 3335 tablet.cc:2378] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 07:57:59.995628 3469 tablet.cc:2378] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 07:58:00.142467 3202 tablet.cc:2378] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:58:00.151062 3493 raft_consensus.cc:491] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:58:00.151477 3493 raft_consensus.cc:513] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.153759 3493 leader_election.cc:290] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f97c4fba2b4eed9456a44b70dfe22f (127.1.121.194:40613), 6ca46ee70eb14f85b481346b9fe0fe91 (127.1.121.195:43425)
I20250815 07:58:00.165046 3289 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d698c0e1196e4050b582f7ae95da9726" candidate_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" is_pre_election: true
I20250815 07:58:00.165211 3423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d698c0e1196e4050b582f7ae95da9726" candidate_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" is_pre_election: true
I20250815 07:58:00.165700 3289 raft_consensus.cc:2466] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7ea35b17ae4980a3ec913ceb19c674 in term 0.
I20250815 07:58:00.165804 3423 raft_consensus.cc:2466] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7ea35b17ae4980a3ec913ceb19c674 in term 0.
I20250815 07:58:00.166718 3090 leader_election.cc:304] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7ea35b17ae4980a3ec913ceb19c674, e2f97c4fba2b4eed9456a44b70dfe22f; no voters:
I20250815 07:58:00.167325 3493 raft_consensus.cc:2802] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 07:58:00.167619 3493 raft_consensus.cc:491] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 07:58:00.167853 3493 raft_consensus.cc:3058] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:00.171864 3493 raft_consensus.cc:513] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.173288 3493 leader_election.cc:290] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [CANDIDATE]: Term 1 election: Requested vote from peers e2f97c4fba2b4eed9456a44b70dfe22f (127.1.121.194:40613), 6ca46ee70eb14f85b481346b9fe0fe91 (127.1.121.195:43425)
I20250815 07:58:00.173887 3289 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d698c0e1196e4050b582f7ae95da9726" candidate_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f"
I20250815 07:58:00.174091 3423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d698c0e1196e4050b582f7ae95da9726" candidate_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca46ee70eb14f85b481346b9fe0fe91"
I20250815 07:58:00.174247 3289 raft_consensus.cc:3058] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:00.174490 3423 raft_consensus.cc:3058] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:00.178429 3289 raft_consensus.cc:2466] T d698c0e1196e4050b582f7ae95da9726 P e2f97c4fba2b4eed9456a44b70dfe22f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ea35b17ae4980a3ec913ceb19c674 in term 1.
I20250815 07:58:00.178824 3423 raft_consensus.cc:2466] T d698c0e1196e4050b582f7ae95da9726 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ea35b17ae4980a3ec913ceb19c674 in term 1.
I20250815 07:58:00.179198 3090 leader_election.cc:304] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7ea35b17ae4980a3ec913ceb19c674, e2f97c4fba2b4eed9456a44b70dfe22f; no voters:
I20250815 07:58:00.179806 3493 raft_consensus.cc:2802] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:58:00.181334 3493 raft_consensus.cc:695] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7ea35b17ae4980a3ec913ceb19c674, State: Running, Role: LEADER
I20250815 07:58:00.181993 3493 consensus_queue.cc:237] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.191845 3010 catalog_manager.cc:5582] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b7ea35b17ae4980a3ec913ceb19c674 (127.1.121.193). New cstate: current_term: 1 leader_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20250815 07:58:00.217741 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:58:00.220666 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7b7ea35b17ae4980a3ec913ceb19c674 to finish bootstrapping
I20250815 07:58:00.231671 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e2f97c4fba2b4eed9456a44b70dfe22f to finish bootstrapping
I20250815 07:58:00.240597 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6ca46ee70eb14f85b481346b9fe0fe91 to finish bootstrapping
I20250815 07:58:00.251385 3010 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36906:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250815 07:58:00.252645 3010 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 07:58:00.266366 3136 tablet_service.cc:1468] Processing CreateTablet for tablet a4999f6eed754f9fb0cdb16176eeeb26 (DEFAULT_TABLE table=TestAnotherTable [id=a78c81df59b142cea82aa18432ab59db]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 07:58:00.266956 3269 tablet_service.cc:1468] Processing CreateTablet for tablet a4999f6eed754f9fb0cdb16176eeeb26 (DEFAULT_TABLE table=TestAnotherTable [id=a78c81df59b142cea82aa18432ab59db]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 07:58:00.267333 3136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4999f6eed754f9fb0cdb16176eeeb26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:00.267562 3403 tablet_service.cc:1468] Processing CreateTablet for tablet a4999f6eed754f9fb0cdb16176eeeb26 (DEFAULT_TABLE table=TestAnotherTable [id=a78c81df59b142cea82aa18432ab59db]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250815 07:58:00.267874 3269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4999f6eed754f9fb0cdb16176eeeb26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:00.268357 3403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4999f6eed754f9fb0cdb16176eeeb26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:00.277539 3489 tablet_bootstrap.cc:492] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91: Bootstrap starting.
I20250815 07:58:00.279620 3488 tablet_bootstrap.cc:492] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674: Bootstrap starting.
I20250815 07:58:00.280440 3487 tablet_bootstrap.cc:492] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f: Bootstrap starting.
I20250815 07:58:00.283110 3489 tablet_bootstrap.cc:654] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:00.283725 3488 tablet_bootstrap.cc:654] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:00.285610 3487 tablet_bootstrap.cc:654] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:00.288573 3489 tablet_bootstrap.cc:492] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91: No bootstrap required, opened a new log
I20250815 07:58:00.288921 3489 ts_tablet_manager.cc:1397] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20250815 07:58:00.288996 3488 tablet_bootstrap.cc:492] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674: No bootstrap required, opened a new log
I20250815 07:58:00.289410 3488 ts_tablet_manager.cc:1397] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250815 07:58:00.291018 3487 tablet_bootstrap.cc:492] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f: No bootstrap required, opened a new log
I20250815 07:58:00.290822 3489 raft_consensus.cc:357] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.291378 3489 raft_consensus.cc:383] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:00.291474 3487 ts_tablet_manager.cc:1397] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.001s
I20250815 07:58:00.291700 3489 raft_consensus.cc:738] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca46ee70eb14f85b481346b9fe0fe91, State: Initialized, Role: FOLLOWER
I20250815 07:58:00.292317 3489 consensus_queue.cc:260] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.291636 3488 raft_consensus.cc:357] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.294158 3489 ts_tablet_manager.cc:1428] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250815 07:58:00.294270 3488 raft_consensus.cc:383] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:00.294525 3488 raft_consensus.cc:738] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7ea35b17ae4980a3ec913ceb19c674, State: Initialized, Role: FOLLOWER
I20250815 07:58:00.294329 3487 raft_consensus.cc:357] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.295099 3487 raft_consensus.cc:383] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:00.295336 3487 raft_consensus.cc:738] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f97c4fba2b4eed9456a44b70dfe22f, State: Initialized, Role: FOLLOWER
I20250815 07:58:00.295082 3488 consensus_queue.cc:260] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.295940 3487 consensus_queue.cc:260] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.297787 3487 ts_tablet_manager.cc:1428] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250815 07:58:00.296700 3488 ts_tablet_manager.cc:1428] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250815 07:58:00.514670 3493 raft_consensus.cc:491] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:58:00.514993 3493 raft_consensus.cc:513] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.516264 3493 leader_election.cc:290] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f97c4fba2b4eed9456a44b70dfe22f (127.1.121.194:40613), 6ca46ee70eb14f85b481346b9fe0fe91 (127.1.121.195:43425)
I20250815 07:58:00.516952 3289 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4999f6eed754f9fb0cdb16176eeeb26" candidate_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" is_pre_election: true
I20250815 07:58:00.517143 3423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4999f6eed754f9fb0cdb16176eeeb26" candidate_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" is_pre_election: true
I20250815 07:58:00.517362 3289 raft_consensus.cc:2466] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7ea35b17ae4980a3ec913ceb19c674 in term 0.
I20250815 07:58:00.517581 3423 raft_consensus.cc:2466] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b7ea35b17ae4980a3ec913ceb19c674 in term 0.
I20250815 07:58:00.518146 3090 leader_election.cc:304] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7ea35b17ae4980a3ec913ceb19c674, e2f97c4fba2b4eed9456a44b70dfe22f; no voters:
I20250815 07:58:00.518669 3493 raft_consensus.cc:2802] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 07:58:00.518975 3493 raft_consensus.cc:491] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 07:58:00.519271 3493 raft_consensus.cc:3058] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:00.522872 3493 raft_consensus.cc:513] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.523988 3493 leader_election.cc:290] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [CANDIDATE]: Term 1 election: Requested vote from peers e2f97c4fba2b4eed9456a44b70dfe22f (127.1.121.194:40613), 6ca46ee70eb14f85b481346b9fe0fe91 (127.1.121.195:43425)
I20250815 07:58:00.524637 3289 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4999f6eed754f9fb0cdb16176eeeb26" candidate_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f"
I20250815 07:58:00.524835 3423 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a4999f6eed754f9fb0cdb16176eeeb26" candidate_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ca46ee70eb14f85b481346b9fe0fe91"
I20250815 07:58:00.525092 3289 raft_consensus.cc:3058] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:00.525254 3423 raft_consensus.cc:3058] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:00.530697 3289 raft_consensus.cc:2466] T a4999f6eed754f9fb0cdb16176eeeb26 P e2f97c4fba2b4eed9456a44b70dfe22f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ea35b17ae4980a3ec913ceb19c674 in term 1.
I20250815 07:58:00.530762 3423 raft_consensus.cc:2466] T a4999f6eed754f9fb0cdb16176eeeb26 P 6ca46ee70eb14f85b481346b9fe0fe91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b7ea35b17ae4980a3ec913ceb19c674 in term 1.
I20250815 07:58:00.531517 3090 leader_election.cc:304] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b7ea35b17ae4980a3ec913ceb19c674, e2f97c4fba2b4eed9456a44b70dfe22f; no voters:
I20250815 07:58:00.532032 3493 raft_consensus.cc:2802] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:58:00.532371 3493 raft_consensus.cc:695] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [term 1 LEADER]: Becoming Leader. State: Replica: 7b7ea35b17ae4980a3ec913ceb19c674, State: Running, Role: LEADER
I20250815 07:58:00.532826 3493 consensus_queue.cc:237] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } }
I20250815 07:58:00.538424 3013 catalog_manager.cc:5582] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b7ea35b17ae4980a3ec913ceb19c674 (127.1.121.193). New cstate: current_term: 1 leader_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b7ea35b17ae4980a3ec913ceb19c674" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 42137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20250815 07:58:00.624439 3493 consensus_queue.cc:1035] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 07:58:00.642779 3498 consensus_queue.cc:1035] T d698c0e1196e4050b582f7ae95da9726 P 7b7ea35b17ae4980a3ec913ceb19c674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 07:58:00.836143 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:00.836601 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:00.862735 3505 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250815 07:58:00.911543 3498 consensus_queue.cc:1035] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f97c4fba2b4eed9456a44b70dfe22f" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 07:58:00.931597 3493 consensus_queue.cc:1035] T a4999f6eed754f9fb0cdb16176eeeb26 P 7b7ea35b17ae4980a3ec913ceb19c674 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ca46ee70eb14f85b481346b9fe0fe91" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 07:58:01.990761 3505 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.094s user 0.443s sys 0.642s
W20250815 07:58:01.991156 3505 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.094s user 0.443s sys 0.642s
W20250815 07:58:03.339145 3526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:03.339681 3526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:03.366339 3526 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 07:58:04.497100 3526 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.096s user 0.421s sys 0.674s
W20250815 07:58:04.497408 3526 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.096s user 0.421s sys 0.674s
W20250815 07:58:05.827181 3541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:05.827708 3541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:05.854697 3541 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 07:58:06.988997 3541 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.100s user 0.418s sys 0.680s
W20250815 07:58:06.989359 3541 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.100s user 0.418s sys 0.680s
W20250815 07:58:08.323846 3555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:08.324347 3555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:08.350476 3555 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 07:58:09.475777 3555 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.091s user 0.000s sys 0.001s
W20250815 07:58:09.476054 3555 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.091s user 0.002s sys 0.002s
I20250815 07:58:10.539850 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 3072
I20250815 07:58:10.564626 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 3205
I20250815 07:58:10.585336 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 3338
I20250815 07:58:10.606211 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 2980
2025-08-15T07:58:10Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18335 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250815 07:58:10.659386 1511 test_util.cc:276] Using random seed: -1069033440
I20250815 07:58:10.662998 1511 ts_itest-base.cc:115] Starting cluster with:
I20250815 07:58:10.663151 1511 ts_itest-base.cc:116] --------------
I20250815 07:58:10.663285 1511 ts_itest-base.cc:117] 3 tablet servers
I20250815 07:58:10.663400 1511 ts_itest-base.cc:118] 3 replicas per TS
I20250815 07:58:10.663522 1511 ts_itest-base.cc:119] --------------
2025-08-15T07:58:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T07:58:10Z Disabled control of system clock
I20250815 07:58:10.692984 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:37999
--webserver_interface=127.1.121.254
--webserver_port=0
--builtin_ntp_servers=127.1.121.212:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:37999
--superuser_acl=no-such-user with env {}
W20250815 07:58:10.946588 3577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:10.947037 3577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:10.947420 3577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:10.973506 3577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:58:10.973737 3577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:10.973910 3577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:58:10.974076 3577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:58:11.005460 3577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:37999
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:11.006511 3577 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:11.007913 3577 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:11.017374 3583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:11.018410 3584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:12.030822 3585 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 07:58:12.031965 3586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:12.031998 3577 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:12.034565 3577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:12.036818 3577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:12.038120 3577 hybrid_clock.cc:648] HybridClock initialized: now 1755244692038061 us; error 58 us; skew 500 ppm
I20250815 07:58:12.038797 3577 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:12.044301 3577 webserver.cc:480] Webserver started at http://127.1.121.254:43639/ using document root <none> and password file <none>
I20250815 07:58:12.045083 3577 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:12.045279 3577 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:12.045681 3577 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:58:12.049461 3577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "11ddac7a67aa48d790bc771b91911dee"
format_stamp: "Formatted at 2025-08-15 07:58:12 on dist-test-slave-dlgj"
I20250815 07:58:12.050480 3577 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "11ddac7a67aa48d790bc771b91911dee"
format_stamp: "Formatted at 2025-08-15 07:58:12 on dist-test-slave-dlgj"
I20250815 07:58:12.056689 3577 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250815 07:58:12.061486 3593 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:12.062325 3577 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250815 07:58:12.062592 3577 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "11ddac7a67aa48d790bc771b91911dee"
format_stamp: "Formatted at 2025-08-15 07:58:12 on dist-test-slave-dlgj"
I20250815 07:58:12.062861 3577 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:12.107189 3577 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:12.108474 3577 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:12.108852 3577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:12.173651 3577 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:37999
I20250815 07:58:12.173702 3644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:37999 every 8 connection(s)
I20250815 07:58:12.176347 3577 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:58:12.178433 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 3577
I20250815 07:58:12.178967 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 07:58:12.182425 3645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:12.203857 3645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee: Bootstrap starting.
I20250815 07:58:12.208461 3645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:12.209878 3645 log.cc:826] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:12.214046 3645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee: No bootstrap required, opened a new log
I20250815 07:58:12.229571 3645 raft_consensus.cc:357] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ddac7a67aa48d790bc771b91911dee" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 37999 } }
I20250815 07:58:12.230161 3645 raft_consensus.cc:383] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:12.230410 3645 raft_consensus.cc:738] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ddac7a67aa48d790bc771b91911dee, State: Initialized, Role: FOLLOWER
I20250815 07:58:12.231037 3645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ddac7a67aa48d790bc771b91911dee" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 37999 } }
I20250815 07:58:12.231500 3645 raft_consensus.cc:397] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:12.231729 3645 raft_consensus.cc:491] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:12.231995 3645 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:12.235486 3645 raft_consensus.cc:513] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ddac7a67aa48d790bc771b91911dee" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 37999 } }
I20250815 07:58:12.236198 3645 leader_election.cc:304] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11ddac7a67aa48d790bc771b91911dee; no voters:
I20250815 07:58:12.237704 3645 leader_election.cc:290] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:58:12.238241 3650 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:58:12.240451 3650 raft_consensus.cc:695] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [term 1 LEADER]: Becoming Leader. State: Replica: 11ddac7a67aa48d790bc771b91911dee, State: Running, Role: LEADER
I20250815 07:58:12.241174 3645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:58:12.241104 3650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ddac7a67aa48d790bc771b91911dee" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 37999 } }
I20250815 07:58:12.250866 3651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11ddac7a67aa48d790bc771b91911dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ddac7a67aa48d790bc771b91911dee" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 37999 } } }
I20250815 07:58:12.251492 3651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:12.250892 3652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11ddac7a67aa48d790bc771b91911dee. Latest consensus state: current_term: 1 leader_uuid: "11ddac7a67aa48d790bc771b91911dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ddac7a67aa48d790bc771b91911dee" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 37999 } } }
I20250815 07:58:12.251986 3652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:12.254182 3658 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:58:12.263782 3658 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:58:12.277487 3658 catalog_manager.cc:1349] Generated new cluster ID: 9dfe9631e7b04c31830a89392054646b
I20250815 07:58:12.277729 3658 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:58:12.301098 3658 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 07:58:12.302310 3658 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:58:12.314643 3658 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 11ddac7a67aa48d790bc771b91911dee: Generated new TSK 0
I20250815 07:58:12.315366 3658 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:58:12.336674 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:0
--local_ip_for_outbound_sockets=127.1.121.193
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:37999
--builtin_ntp_servers=127.1.121.212:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 07:58:12.610569 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:12.611006 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:12.611433 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:12.638796 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:12.639580 3669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:58:12.669776 3669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:37999
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:12.671078 3669 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:12.672633 3669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:12.684131 3675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:12.687206 3678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:12.685526 3676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:13.712343 3677 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 07:58:13.712859 3669 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:13.715742 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:13.718084 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:13.719494 3669 hybrid_clock.cc:648] HybridClock initialized: now 1755244693719451 us; error 56 us; skew 500 ppm
I20250815 07:58:13.720306 3669 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:13.726392 3669 webserver.cc:480] Webserver started at http://127.1.121.193:41995/ using document root <none> and password file <none>
I20250815 07:58:13.727160 3669 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:13.727348 3669 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:13.727749 3669 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:58:13.731639 3669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7d49229ed6bb40a3b568d3818ea3db1c"
format_stamp: "Formatted at 2025-08-15 07:58:13 on dist-test-slave-dlgj"
I20250815 07:58:13.732623 3669 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7d49229ed6bb40a3b568d3818ea3db1c"
format_stamp: "Formatted at 2025-08-15 07:58:13 on dist-test-slave-dlgj"
I20250815 07:58:13.738394 3669 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250815 07:58:13.743268 3685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:13.744118 3669 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250815 07:58:13.744380 3669 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7d49229ed6bb40a3b568d3818ea3db1c"
format_stamp: "Formatted at 2025-08-15 07:58:13 on dist-test-slave-dlgj"
I20250815 07:58:13.744650 3669 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:13.795137 3669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:13.796314 3669 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:13.796665 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:13.798712 3669 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:13.802196 3669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:58:13.802371 3669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:13.802613 3669 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:58:13.802758 3669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:13.926330 3669 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:45363
I20250815 07:58:13.926424 3797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:45363 every 8 connection(s)
I20250815 07:58:13.928696 3669 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:58:13.938735 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 3669
I20250815 07:58:13.939079 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 07:58:13.945031 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:0
--local_ip_for_outbound_sockets=127.1.121.194
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:37999
--builtin_ntp_servers=127.1.121.212:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:13.947141 3798 heartbeater.cc:344] Connected to a master server at 127.1.121.254:37999
I20250815 07:58:13.947592 3798 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:13.948752 3798 heartbeater.cc:507] Master 127.1.121.254:37999 requested a full tablet report, sending...
I20250815 07:58:13.951205 3610 ts_manager.cc:194] Registered new tserver with Master: 7d49229ed6bb40a3b568d3818ea3db1c (127.1.121.193:45363)
I20250815 07:58:13.953454 3610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:60031
W20250815 07:58:14.214082 3802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:14.214485 3802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:14.214850 3802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:14.240826 3802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:14.241518 3802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:58:14.270195 3802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:37999
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:14.271276 3802 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:14.272590 3802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:14.281951 3808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:14.956506 3798 heartbeater.cc:499] Master 127.1.121.254:37999 was elected leader, sending a full tablet report...
W20250815 07:58:14.283623 3809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:15.686036 3807 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3802
W20250815 07:58:15.794680 3802 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.512s user 0.583s sys 0.928s
W20250815 07:58:15.795751 3802 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.513s user 0.583s sys 0.928s
W20250815 07:58:15.796188 3810 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1511 milliseconds
W20250815 07:58:15.796855 3811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:15.796813 3802 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:15.799499 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:15.801410 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:15.802706 3802 hybrid_clock.cc:648] HybridClock initialized: now 1755244695802667 us; error 38 us; skew 500 ppm
I20250815 07:58:15.803349 3802 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:15.808410 3802 webserver.cc:480] Webserver started at http://127.1.121.194:44477/ using document root <none> and password file <none>
I20250815 07:58:15.809175 3802 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:15.809361 3802 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:15.809792 3802 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:58:15.813526 3802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "eb49de64deb4471ab7be95339aa60a72"
format_stamp: "Formatted at 2025-08-15 07:58:15 on dist-test-slave-dlgj"
I20250815 07:58:15.814435 3802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "eb49de64deb4471ab7be95339aa60a72"
format_stamp: "Formatted at 2025-08-15 07:58:15 on dist-test-slave-dlgj"
I20250815 07:58:15.820362 3802 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20250815 07:58:15.825006 3818 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:15.825819 3802 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250815 07:58:15.826069 3802 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "eb49de64deb4471ab7be95339aa60a72"
format_stamp: "Formatted at 2025-08-15 07:58:15 on dist-test-slave-dlgj"
I20250815 07:58:15.826340 3802 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:15.885739 3802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:15.886889 3802 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:15.887243 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:15.889295 3802 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:15.893301 3802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:58:15.893479 3802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:15.893688 3802 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:58:15.893826 3802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:16.011349 3802 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:43637
I20250815 07:58:16.011441 3930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:43637 every 8 connection(s)
I20250815 07:58:16.013643 3802 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 07:58:16.017659 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 3802
I20250815 07:58:16.017999 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 07:58:16.023931 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:0
--local_ip_for_outbound_sockets=127.1.121.195
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:37999
--builtin_ntp_servers=127.1.121.212:46565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:16.038214 3931 heartbeater.cc:344] Connected to a master server at 127.1.121.254:37999
I20250815 07:58:16.038566 3931 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:16.039404 3931 heartbeater.cc:507] Master 127.1.121.254:37999 requested a full tablet report, sending...
I20250815 07:58:16.041294 3610 ts_manager.cc:194] Registered new tserver with Master: eb49de64deb4471ab7be95339aa60a72 (127.1.121.194:43637)
I20250815 07:58:16.042339 3610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:37341
W20250815 07:58:16.294114 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:16.294570 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:16.294996 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:16.327704 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:16.329316 3935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:58:16.369705 3935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:46565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:37999
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:16.370841 3935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:16.372193 3935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:16.381986 3941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:17.044900 3931 heartbeater.cc:499] Master 127.1.121.254:37999 was elected leader, sending a full tablet report...
W20250815 07:58:17.786381 3940 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3935
W20250815 07:58:17.850297 3940 kernel_stack_watchdog.cc:198] Thread 3935 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 07:58:17.850396 3935 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.467s user 0.542s sys 0.920s
W20250815 07:58:16.383322 3942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:17.851274 3935 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.468s user 0.543s sys 0.920s
W20250815 07:58:17.852882 3944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:17.855057 3935 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 07:58:17.855005 3943 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1467 milliseconds
I20250815 07:58:17.856200 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:17.857969 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:17.859253 3935 hybrid_clock.cc:648] HybridClock initialized: now 1755244697859216 us; error 40 us; skew 500 ppm
I20250815 07:58:17.859938 3935 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:17.865216 3935 webserver.cc:480] Webserver started at http://127.1.121.195:41151/ using document root <none> and password file <none>
I20250815 07:58:17.865955 3935 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:17.866107 3935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:17.866446 3935 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:58:17.870327 3935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b26cd06219aa42e795e6bb1ed7477e2d"
format_stamp: "Formatted at 2025-08-15 07:58:17 on dist-test-slave-dlgj"
I20250815 07:58:17.871266 3935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b26cd06219aa42e795e6bb1ed7477e2d"
format_stamp: "Formatted at 2025-08-15 07:58:17 on dist-test-slave-dlgj"
I20250815 07:58:17.877225 3935 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250815 07:58:17.882041 3951 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:17.882856 3935 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250815 07:58:17.883092 3935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b26cd06219aa42e795e6bb1ed7477e2d"
format_stamp: "Formatted at 2025-08-15 07:58:17 on dist-test-slave-dlgj"
I20250815 07:58:17.883328 3935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:17.927315 3935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:17.928690 3935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:17.929121 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:17.931828 3935 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:17.935184 3935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:58:17.935364 3935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:17.935590 3935 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:58:17.935726 3935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:18.056445 3935 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:41237
I20250815 07:58:18.056542 4063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:41237 every 8 connection(s)
I20250815 07:58:18.059010 3935 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 07:58:18.063737 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 3935
I20250815 07:58:18.064131 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 07:58:18.078092 4064 heartbeater.cc:344] Connected to a master server at 127.1.121.254:37999
I20250815 07:58:18.078452 4064 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:18.079279 4064 heartbeater.cc:507] Master 127.1.121.254:37999 requested a full tablet report, sending...
I20250815 07:58:18.081202 3610 ts_manager.cc:194] Registered new tserver with Master: b26cd06219aa42e795e6bb1ed7477e2d (127.1.121.195:41237)
I20250815 07:58:18.082410 3610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:40839
I20250815 07:58:18.083456 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:58:18.109395 3610 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40654:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250815 07:58:18.126212 3610 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 07:58:18.180181 3866 tablet_service.cc:1468] Processing CreateTablet for tablet 9581574b7a0948e8ac38b4f128fddcd2 (DEFAULT_TABLE table=TestTable [id=92b29fdd904c4e049e795a5d04ec201a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:58:18.181965 3866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9581574b7a0948e8ac38b4f128fddcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:18.183184 3999 tablet_service.cc:1468] Processing CreateTablet for tablet 9581574b7a0948e8ac38b4f128fddcd2 (DEFAULT_TABLE table=TestTable [id=92b29fdd904c4e049e795a5d04ec201a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:58:18.183249 3733 tablet_service.cc:1468] Processing CreateTablet for tablet 9581574b7a0948e8ac38b4f128fddcd2 (DEFAULT_TABLE table=TestTable [id=92b29fdd904c4e049e795a5d04ec201a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:58:18.185004 3999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9581574b7a0948e8ac38b4f128fddcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:18.187162 3733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9581574b7a0948e8ac38b4f128fddcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:18.214702 4083 tablet_bootstrap.cc:492] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72: Bootstrap starting.
I20250815 07:58:18.215337 4084 tablet_bootstrap.cc:492] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d: Bootstrap starting.
I20250815 07:58:18.222018 4083 tablet_bootstrap.cc:654] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:18.223856 4084 tablet_bootstrap.cc:654] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:18.224159 4083 log.cc:826] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:18.224519 4085 tablet_bootstrap.cc:492] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c: Bootstrap starting.
I20250815 07:58:18.225978 4084 log.cc:826] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:18.231709 4085 tablet_bootstrap.cc:654] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:18.233585 4085 log.cc:826] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:18.235095 4083 tablet_bootstrap.cc:492] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72: No bootstrap required, opened a new log
I20250815 07:58:18.235554 4083 ts_tablet_manager.cc:1397] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20250815 07:58:18.236766 4084 tablet_bootstrap.cc:492] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d: No bootstrap required, opened a new log
I20250815 07:58:18.237244 4084 ts_tablet_manager.cc:1397] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.008s
I20250815 07:58:18.245245 4085 tablet_bootstrap.cc:492] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c: No bootstrap required, opened a new log
I20250815 07:58:18.245589 4085 ts_tablet_manager.cc:1397] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20250815 07:58:18.258631 4083 raft_consensus.cc:357] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
I20250815 07:58:18.259392 4083 raft_consensus.cc:383] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:18.259670 4083 raft_consensus.cc:738] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb49de64deb4471ab7be95339aa60a72, State: Initialized, Role: FOLLOWER
I20250815 07:58:18.260574 4083 consensus_queue.cc:260] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
I20250815 07:58:18.261076 4084 raft_consensus.cc:357] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
I20250815 07:58:18.261404 4085 raft_consensus.cc:357] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
I20250815 07:58:18.261847 4084 raft_consensus.cc:383] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:18.262032 4085 raft_consensus.cc:383] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:18.262215 4084 raft_consensus.cc:738] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26cd06219aa42e795e6bb1ed7477e2d, State: Initialized, Role: FOLLOWER
I20250815 07:58:18.262393 4085 raft_consensus.cc:738] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d49229ed6bb40a3b568d3818ea3db1c, State: Initialized, Role: FOLLOWER
I20250815 07:58:18.263264 4085 consensus_queue.cc:260] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
I20250815 07:58:18.263216 4084 consensus_queue.cc:260] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
I20250815 07:58:18.264999 4083 ts_tablet_manager.cc:1428] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72: Time spent starting tablet: real 0.029s user 0.018s sys 0.009s
I20250815 07:58:18.267880 4064 heartbeater.cc:499] Master 127.1.121.254:37999 was elected leader, sending a full tablet report...
I20250815 07:58:18.268776 4085 ts_tablet_manager.cc:1428] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c: Time spent starting tablet: real 0.023s user 0.020s sys 0.004s
I20250815 07:58:18.267907 4084 ts_tablet_manager.cc:1428] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d: Time spent starting tablet: real 0.030s user 0.018s sys 0.009s
W20250815 07:58:18.274107 3932 tablet.cc:2378] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:58:18.279780 4089 raft_consensus.cc:491] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:58:18.280205 4089 raft_consensus.cc:513] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
I20250815 07:58:18.282171 4089 leader_election.cc:290] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d49229ed6bb40a3b568d3818ea3db1c (127.1.121.193:45363), b26cd06219aa42e795e6bb1ed7477e2d (127.1.121.195:41237)
I20250815 07:58:18.291355 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9581574b7a0948e8ac38b4f128fddcd2" candidate_uuid: "eb49de64deb4471ab7be95339aa60a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" is_pre_election: true
I20250815 07:58:18.291986 3753 raft_consensus.cc:2466] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb49de64deb4471ab7be95339aa60a72 in term 0.
I20250815 07:58:18.292918 3819 leader_election.cc:304] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d49229ed6bb40a3b568d3818ea3db1c, eb49de64deb4471ab7be95339aa60a72; no voters:
I20250815 07:58:18.293613 4089 raft_consensus.cc:2802] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 07:58:18.293529 4019 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9581574b7a0948e8ac38b4f128fddcd2" candidate_uuid: "eb49de64deb4471ab7be95339aa60a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" is_pre_election: true
I20250815 07:58:18.293844 4089 raft_consensus.cc:491] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 07:58:18.294083 4089 raft_consensus.cc:3058] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:18.294163 4019 raft_consensus.cc:2466] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb49de64deb4471ab7be95339aa60a72 in term 0.
I20250815 07:58:18.298123 4089 raft_consensus.cc:513] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
I20250815 07:58:18.299194 4089 leader_election.cc:290] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [CANDIDATE]: Term 1 election: Requested vote from peers 7d49229ed6bb40a3b568d3818ea3db1c (127.1.121.193:45363), b26cd06219aa42e795e6bb1ed7477e2d (127.1.121.195:41237)
I20250815 07:58:18.299784 3753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9581574b7a0948e8ac38b4f128fddcd2" candidate_uuid: "eb49de64deb4471ab7be95339aa60a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d49229ed6bb40a3b568d3818ea3db1c"
I20250815 07:58:18.299978 4019 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9581574b7a0948e8ac38b4f128fddcd2" candidate_uuid: "eb49de64deb4471ab7be95339aa60a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26cd06219aa42e795e6bb1ed7477e2d"
I20250815 07:58:18.300175 3753 raft_consensus.cc:3058] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:18.300429 4019 raft_consensus.cc:3058] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:18.303871 3753 raft_consensus.cc:2466] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49de64deb4471ab7be95339aa60a72 in term 1.
I20250815 07:58:18.304046 4019 raft_consensus.cc:2466] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb49de64deb4471ab7be95339aa60a72 in term 1.
I20250815 07:58:18.304550 3819 leader_election.cc:304] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d49229ed6bb40a3b568d3818ea3db1c, eb49de64deb4471ab7be95339aa60a72; no voters:
I20250815 07:58:18.305012 4089 raft_consensus.cc:2802] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:58:18.306339 4089 raft_consensus.cc:695] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [term 1 LEADER]: Becoming Leader. State: Replica: eb49de64deb4471ab7be95339aa60a72, State: Running, Role: LEADER
I20250815 07:58:18.307047 4089 consensus_queue.cc:237] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } }
W20250815 07:58:18.313254 4065 tablet.cc:2378] T 9581574b7a0948e8ac38b4f128fddcd2 P b26cd06219aa42e795e6bb1ed7477e2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:58:18.316660 3610 catalog_manager.cc:5582] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb49de64deb4471ab7be95339aa60a72 (127.1.121.194). New cstate: current_term: 1 leader_uuid: "eb49de64deb4471ab7be95339aa60a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb49de64deb4471ab7be95339aa60a72" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 43637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 } health_report { overall_health: UNKNOWN } } }
I20250815 07:58:18.335178 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:58:18.337985 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7d49229ed6bb40a3b568d3818ea3db1c to finish bootstrapping
I20250815 07:58:18.350111 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver eb49de64deb4471ab7be95339aa60a72 to finish bootstrapping
I20250815 07:58:18.359608 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b26cd06219aa42e795e6bb1ed7477e2d to finish bootstrapping
W20250815 07:58:18.436990 3799 tablet.cc:2378] T 9581574b7a0948e8ac38b4f128fddcd2 P 7d49229ed6bb40a3b568d3818ea3db1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:58:18.725713 4089 consensus_queue.cc:1035] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b26cd06219aa42e795e6bb1ed7477e2d" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 07:58:18.760509 4094 consensus_queue.cc:1035] T 9581574b7a0948e8ac38b4f128fddcd2 P eb49de64deb4471ab7be95339aa60a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d49229ed6bb40a3b568d3818ea3db1c" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 45363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250815 07:58:19.793634 3608 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:40684
I20250815 07:58:20.825783 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 3669
W20250815 07:58:20.847054 3819 connection.cc:537] client connection to 127.1.121.193:45363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250815 07:58:20.847210 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 3802
I20250815 07:58:20.869145 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 3935
I20250815 07:58:20.889775 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 3577
2025-08-15T07:58:20Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10285 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250815 07:58:20.944715 1511 test_util.cc:276] Using random seed: -1058748118
I20250815 07:58:20.948271 1511 ts_itest-base.cc:115] Starting cluster with:
I20250815 07:58:20.948410 1511 ts_itest-base.cc:116] --------------
I20250815 07:58:20.948513 1511 ts_itest-base.cc:117] 3 tablet servers
I20250815 07:58:20.948609 1511 ts_itest-base.cc:118] 3 replicas per TS
I20250815 07:58:20.948705 1511 ts_itest-base.cc:119] --------------
2025-08-15T07:58:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T07:58:20Z Disabled control of system clock
I20250815 07:58:20.979043 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46565
--webserver_interface=127.1.121.254
--webserver_port=0
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:46565 with env {}
W20250815 07:58:21.241364 4129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:21.241822 4129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:21.242169 4129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:21.268889 4129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:58:21.269120 4129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:21.269302 4129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:58:21.269470 4129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:58:21.298942 4129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:46565
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:21.300086 4129 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:21.301499 4129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:21.310037 4135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:22.714147 4134 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 4129
W20250815 07:58:22.881815 4129 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.571s user 0.580s sys 0.991s
W20250815 07:58:22.882138 4129 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.571s user 0.580s sys 0.991s
W20250815 07:58:21.311545 4136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:22.883436 4137 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1571 milliseconds
I20250815 07:58:22.884172 4129 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 07:58:22.884208 4138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:22.887107 4129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:22.889302 4129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:22.890594 4129 hybrid_clock.cc:648] HybridClock initialized: now 1755244702890560 us; error 43 us; skew 500 ppm
I20250815 07:58:22.891350 4129 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:22.896590 4129 webserver.cc:480] Webserver started at http://127.1.121.254:33975/ using document root <none> and password file <none>
I20250815 07:58:22.897395 4129 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:22.897594 4129 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:22.898015 4129 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:58:22.902019 4129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "70e191f458824dfdad9ad80b91f413a6"
format_stamp: "Formatted at 2025-08-15 07:58:22 on dist-test-slave-dlgj"
I20250815 07:58:22.902980 4129 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "70e191f458824dfdad9ad80b91f413a6"
format_stamp: "Formatted at 2025-08-15 07:58:22 on dist-test-slave-dlgj"
I20250815 07:58:22.909179 4129 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250815 07:58:22.913890 4145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:22.914716 4129 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250815 07:58:22.915000 4129 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "70e191f458824dfdad9ad80b91f413a6"
format_stamp: "Formatted at 2025-08-15 07:58:22 on dist-test-slave-dlgj"
I20250815 07:58:22.915288 4129 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:22.958429 4129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:22.959674 4129 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:22.960111 4129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:23.019740 4129 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:46565
I20250815 07:58:23.019790 4196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:46565 every 8 connection(s)
I20250815 07:58:23.022239 4129 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:58:23.026575 4197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:23.030737 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 4129
I20250815 07:58:23.031093 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 07:58:23.047708 4197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap starting.
I20250815 07:58:23.052456 4197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:23.054373 4197 log.cc:826] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:23.058336 4197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: No bootstrap required, opened a new log
I20250815 07:58:23.072299 4197 raft_consensus.cc:357] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:23.072813 4197 raft_consensus.cc:383] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:23.073004 4197 raft_consensus.cc:738] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e191f458824dfdad9ad80b91f413a6, State: Initialized, Role: FOLLOWER
I20250815 07:58:23.073531 4197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:23.073951 4197 raft_consensus.cc:397] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:23.074172 4197 raft_consensus.cc:491] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:23.074432 4197 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:23.077736 4197 raft_consensus.cc:513] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:23.078341 4197 leader_election.cc:304] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e191f458824dfdad9ad80b91f413a6; no voters:
I20250815 07:58:23.079721 4197 leader_election.cc:290] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:58:23.080338 4202 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:58:23.082222 4202 raft_consensus.cc:695] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 1 LEADER]: Becoming Leader. State: Replica: 70e191f458824dfdad9ad80b91f413a6, State: Running, Role: LEADER
I20250815 07:58:23.082931 4202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:23.083837 4197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:58:23.091430 4203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e191f458824dfdad9ad80b91f413a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } } }
I20250815 07:58:23.091528 4204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e191f458824dfdad9ad80b91f413a6. Latest consensus state: current_term: 1 leader_uuid: "70e191f458824dfdad9ad80b91f413a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } } }
I20250815 07:58:23.092190 4203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:23.092192 4204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:23.096418 4211 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:58:23.106925 4211 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:58:23.108733 4219 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250815 07:58:23.121085 4211 catalog_manager.cc:1349] Generated new cluster ID: 0927bf0b94ce495f8be57a0848bb40f8
I20250815 07:58:23.121415 4211 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:58:23.137591 4211 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 07:58:23.139154 4211 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:58:23.176571 4211 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Generated new TSK 0
I20250815 07:58:23.177562 4211 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:58:23.193028 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:0
--local_ip_for_outbound_sockets=127.1.121.193
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46565
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 07:58:23.459856 4221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:23.460479 4221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:23.460968 4221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:23.487902 4221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:23.488651 4221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:58:23.519023 4221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:23.520279 4221 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:23.521766 4221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:23.532594 4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:23.533444 4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:24.810542 4229 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1275 milliseconds
W20250815 07:58:24.811262 4230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:24.811901 4221 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.278s user 0.349s sys 0.897s
W20250815 07:58:24.812238 4221 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.279s user 0.353s sys 0.897s
I20250815 07:58:24.812485 4221 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:24.813812 4221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:24.816459 4221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:24.817924 4221 hybrid_clock.cc:648] HybridClock initialized: now 1755244704817848 us; error 66 us; skew 500 ppm
I20250815 07:58:24.818912 4221 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:24.827289 4221 webserver.cc:480] Webserver started at http://127.1.121.193:44205/ using document root <none> and password file <none>
I20250815 07:58:24.828531 4221 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:24.828786 4221 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:24.829329 4221 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:58:24.835664 4221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95"
format_stamp: "Formatted at 2025-08-15 07:58:24 on dist-test-slave-dlgj"
I20250815 07:58:24.837059 4221 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95"
format_stamp: "Formatted at 2025-08-15 07:58:24 on dist-test-slave-dlgj"
I20250815 07:58:24.845999 4221 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250815 07:58:24.852890 4237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:24.853933 4221 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250815 07:58:24.854274 4221 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95"
format_stamp: "Formatted at 2025-08-15 07:58:24 on dist-test-slave-dlgj"
I20250815 07:58:24.854645 4221 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:24.925523 4221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:24.927287 4221 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:24.927788 4221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:24.930863 4221 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:24.935925 4221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:58:24.936123 4221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250815 07:58:24.936352 4221 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:58:24.936482 4221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:25.084635 4221 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:37599
I20250815 07:58:25.084793 4349 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:37599 every 8 connection(s)
I20250815 07:58:25.086972 4221 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:58:25.092381 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 4221
I20250815 07:58:25.092795 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 07:58:25.099581 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:0
--local_ip_for_outbound_sockets=127.1.121.194
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46565
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:25.119330 4350 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:25.119692 4350 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:25.120635 4350 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:25.122776 4162 ts_manager.cc:194] Registered new tserver with Master: 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193:37599)
I20250815 07:58:25.124495 4162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:37435
W20250815 07:58:25.386365 4354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:25.386780 4354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:25.387209 4354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:25.413679 4354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:25.414386 4354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:58:25.443074 4354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:25.444240 4354 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:25.445641 4354 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:25.456462 4360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:26.127247 4350 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
W20250815 07:58:26.859053 4359 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4354
W20250815 07:58:26.988619 4354 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.532s user 0.599s sys 0.933s
W20250815 07:58:26.988933 4354 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.533s user 0.599s sys 0.933s
W20250815 07:58:25.456847 4361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:26.990708 4363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:26.992878 4362 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1532 milliseconds
I20250815 07:58:26.992986 4354 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:26.994201 4354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:26.996068 4354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:26.997474 4354 hybrid_clock.cc:648] HybridClock initialized: now 1755244706997427 us; error 37 us; skew 500 ppm
I20250815 07:58:26.998165 4354 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:27.003542 4354 webserver.cc:480] Webserver started at http://127.1.121.194:42825/ using document root <none> and password file <none>
I20250815 07:58:27.004364 4354 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:27.004606 4354 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:27.005054 4354 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:58:27.008849 4354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "eb9b068c8b91482fb5236dccf294f0d4"
format_stamp: "Formatted at 2025-08-15 07:58:27 on dist-test-slave-dlgj"
I20250815 07:58:27.009750 4354 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "eb9b068c8b91482fb5236dccf294f0d4"
format_stamp: "Formatted at 2025-08-15 07:58:27 on dist-test-slave-dlgj"
I20250815 07:58:27.015625 4354 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250815 07:58:27.020511 4370 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:27.021488 4354 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250815 07:58:27.021776 4354 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "eb9b068c8b91482fb5236dccf294f0d4"
format_stamp: "Formatted at 2025-08-15 07:58:27 on dist-test-slave-dlgj"
I20250815 07:58:27.022034 4354 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:27.080186 4354 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:27.081501 4354 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:27.081873 4354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:27.084108 4354 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:27.087965 4354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:58:27.088181 4354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:27.088382 4354 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:58:27.088527 4354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:27.208720 4354 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:37329
I20250815 07:58:27.208802 4482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:37329 every 8 connection(s)
I20250815 07:58:27.211242 4354 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 07:58:27.218046 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 4354
I20250815 07:58:27.218438 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 07:58:27.225476 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:0
--local_ip_for_outbound_sockets=127.1.121.195
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46565
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:27.234576 4483 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:27.234943 4483 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:27.235839 4483 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:27.237728 4162 ts_manager.cc:194] Registered new tserver with Master: eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329)
I20250815 07:58:27.238854 4162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:51081
W20250815 07:58:27.486552 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:27.486995 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:27.487437 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:27.514767 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:27.515506 4487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:58:27.545818 4487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:27.546985 4487 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:27.548640 4487 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:27.558379 4493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:28.241700 4483 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
W20250815 07:58:27.559638 4494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:28.962275 4492 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4487
W20250815 07:58:29.064467 4487 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.505s user 0.539s sys 0.965s
W20250815 07:58:29.065205 4487 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.506s user 0.539s sys 0.965s
W20250815 07:58:29.066458 4496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:29.068233 4495 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1506 milliseconds
I20250815 07:58:29.068253 4487 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:29.069236 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:29.070988 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:29.072280 4487 hybrid_clock.cc:648] HybridClock initialized: now 1755244709072262 us; error 36 us; skew 500 ppm
I20250815 07:58:29.072932 4487 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:29.077834 4487 webserver.cc:480] Webserver started at http://127.1.121.195:40185/ using document root <none> and password file <none>
I20250815 07:58:29.078671 4487 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:29.078856 4487 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:29.079231 4487 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:58:29.083770 4487 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee"
format_stamp: "Formatted at 2025-08-15 07:58:29 on dist-test-slave-dlgj"
I20250815 07:58:29.085040 4487 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee"
format_stamp: "Formatted at 2025-08-15 07:58:29 on dist-test-slave-dlgj"
I20250815 07:58:29.093101 4487 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250815 07:58:29.098992 4503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:29.099793 4487 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250815 07:58:29.100113 4487 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee"
format_stamp: "Formatted at 2025-08-15 07:58:29 on dist-test-slave-dlgj"
I20250815 07:58:29.100399 4487 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:29.144073 4487 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:29.145334 4487 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:29.145751 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:29.147962 4487 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:29.151589 4487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:58:29.151784 4487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:29.152052 4487 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:58:29.152197 4487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:29.275991 4487 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:43941
I20250815 07:58:29.276090 4615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:43941 every 8 connection(s)
I20250815 07:58:29.278231 4487 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 07:58:29.282264 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 4487
I20250815 07:58:29.282613 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 07:58:29.301389 4616 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:29.301801 4616 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:29.302637 4616 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:29.304288 4162 ts_manager.cc:194] Registered new tserver with Master: bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941)
I20250815 07:58:29.305285 4162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:37117
I20250815 07:58:29.314579 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:58:29.343820 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:58:29.344300 1511 test_util.cc:276] Using random seed: -1050348514
I20250815 07:58:29.378759 4162 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54858:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 07:58:29.413192 4285 tablet_service.cc:1468] Processing CreateTablet for tablet 3150c850797343f4bbab2cc804e4bc88 (DEFAULT_TABLE table=TestTable [id=01254a56a2f047608bfcd40ddea87eaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:58:29.414536 4285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3150c850797343f4bbab2cc804e4bc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:29.430737 4637 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap starting.
I20250815 07:58:29.435883 4637 tablet_bootstrap.cc:654] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:29.437300 4637 log.cc:826] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:29.440847 4637 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: No bootstrap required, opened a new log
I20250815 07:58:29.441183 4637 ts_tablet_manager.cc:1397] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20250815 07:58:29.455916 4637 raft_consensus.cc:357] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:29.456400 4637 raft_consensus.cc:383] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:29.456593 4637 raft_consensus.cc:738] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Initialized, Role: FOLLOWER
I20250815 07:58:29.457123 4637 consensus_queue.cc:260] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:29.457552 4637 raft_consensus.cc:397] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:29.457770 4637 raft_consensus.cc:491] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:29.458025 4637 raft_consensus.cc:3058] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:29.461624 4637 raft_consensus.cc:513] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:29.462244 4637 leader_election.cc:304] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cb7f777e88841a3a4aa5b37a1ab2b95; no voters:
I20250815 07:58:29.463652 4637 leader_election.cc:290] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:58:29.463953 4639 raft_consensus.cc:2802] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:58:29.467547 4639 raft_consensus.cc:695] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 1 LEADER]: Becoming Leader. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Running, Role: LEADER
I20250815 07:58:29.468613 4637 ts_tablet_manager.cc:1428] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent starting tablet: real 0.027s user 0.024s sys 0.003s
I20250815 07:58:29.468382 4639 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:29.477118 4162 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193). New cstate: current_term: 1 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } }
I20250815 07:58:29.669037 1511 test_util.cc:276] Using random seed: -1050023789
I20250815 07:58:29.686486 4158 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54860:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 07:58:29.711385 4551 tablet_service.cc:1468] Processing CreateTablet for tablet 47b2ca7263944000837d35652b69ec59 (DEFAULT_TABLE table=TestTable1 [id=d1bacea415c54f83b74e5e7b823cd719]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:58:29.712657 4551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b2ca7263944000837d35652b69ec59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:29.728791 4658 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap starting.
I20250815 07:58:29.733388 4658 tablet_bootstrap.cc:654] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:29.735103 4658 log.cc:826] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:29.738539 4658 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: No bootstrap required, opened a new log
I20250815 07:58:29.738823 4658 ts_tablet_manager.cc:1397] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20250815 07:58:29.753307 4658 raft_consensus.cc:357] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } }
I20250815 07:58:29.753844 4658 raft_consensus.cc:383] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:29.754058 4658 raft_consensus.cc:738] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1eea6c51dc4358a6e5cddd3cb203ee, State: Initialized, Role: FOLLOWER
I20250815 07:58:29.754666 4658 consensus_queue.cc:260] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } }
I20250815 07:58:29.755092 4658 raft_consensus.cc:397] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:29.755327 4658 raft_consensus.cc:491] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:29.755594 4658 raft_consensus.cc:3058] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:29.758921 4658 raft_consensus.cc:513] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } }
I20250815 07:58:29.759459 4658 leader_election.cc:304] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1eea6c51dc4358a6e5cddd3cb203ee; no voters:
I20250815 07:58:29.760946 4658 leader_election.cc:290] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:58:29.761263 4660 raft_consensus.cc:2802] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:58:29.763523 4660 raft_consensus.cc:695] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 1 LEADER]: Becoming Leader. State: Replica: bd1eea6c51dc4358a6e5cddd3cb203ee, State: Running, Role: LEADER
I20250815 07:58:29.764242 4616 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
I20250815 07:58:29.764294 4660 consensus_queue.cc:237] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } }
I20250815 07:58:29.764741 4658 ts_tablet_manager.cc:1428] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent starting tablet: real 0.026s user 0.013s sys 0.013s
I20250815 07:58:29.773402 4158 catalog_manager.cc:5582] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee reported cstate change: term changed from 0 to 1, leader changed from <none> to bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195). New cstate: current_term: 1 leader_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } health_report { overall_health: HEALTHY } } }
I20250815 07:58:29.937903 1511 test_util.cc:276] Using random seed: -1049754927
I20250815 07:58:29.955696 4153 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54872:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250815 07:58:29.979923 4418 tablet_service.cc:1468] Processing CreateTablet for tablet cb3af857207e4386ad2828460e9a90db (DEFAULT_TABLE table=TestTable2 [id=4ef17d00c5654caf900bfe447aec6d4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:58:29.981307 4418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb3af857207e4386ad2828460e9a90db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:29.997319 4679 tablet_bootstrap.cc:492] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap starting.
I20250815 07:58:30.001853 4679 tablet_bootstrap.cc:654] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Neither blocks nor log segments found. Creating new log.
I20250815 07:58:30.003551 4679 log.cc:826] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:30.007028 4679 tablet_bootstrap.cc:492] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: No bootstrap required, opened a new log
I20250815 07:58:30.007339 4679 ts_tablet_manager.cc:1397] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250815 07:58:30.021570 4679 raft_consensus.cc:357] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:30.022070 4679 raft_consensus.cc:383] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:58:30.022276 4679 raft_consensus.cc:738] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Initialized, Role: FOLLOWER
I20250815 07:58:30.022980 4679 consensus_queue.cc:260] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:30.023439 4679 raft_consensus.cc:397] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:30.023726 4679 raft_consensus.cc:491] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:30.024056 4679 raft_consensus.cc:3058] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:58:30.027474 4679 raft_consensus.cc:513] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:30.028025 4679 leader_election.cc:304] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9b068c8b91482fb5236dccf294f0d4; no voters:
I20250815 07:58:30.029587 4679 leader_election.cc:290] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:58:30.029946 4681 raft_consensus.cc:2802] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:58:30.033672 4679 ts_tablet_manager.cc:1428] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Time spent starting tablet: real 0.026s user 0.022s sys 0.004s
I20250815 07:58:30.033892 4681 raft_consensus.cc:695] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 1 LEADER]: Becoming Leader. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Running, Role: LEADER
I20250815 07:58:30.034587 4681 consensus_queue.cc:237] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:30.044871 4153 catalog_manager.cc:5582] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194). New cstate: current_term: 1 leader_uuid: "eb9b068c8b91482fb5236dccf294f0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } health_report { overall_health: HEALTHY } } }
I20250815 07:58:30.230443 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 4129
W20250815 07:58:30.494354 4350 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:46565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:46565: connect: Connection refused (error 111)
W20250815 07:58:30.789948 4616 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:46565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:46565: connect: Connection refused (error 111)
W20250815 07:58:31.062572 4483 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:46565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:46565: connect: Connection refused (error 111)
I20250815 07:58:34.429085 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 4221
I20250815 07:58:34.451040 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 4354
I20250815 07:58:34.473508 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 4487
I20250815 07:58:34.498682 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46565
--webserver_interface=127.1.121.254
--webserver_port=33975
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:46565 with env {}
W20250815 07:58:34.759676 4756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:34.760262 4756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:34.760707 4756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:34.786666 4756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:58:34.786926 4756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:34.787135 4756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:58:34.787346 4756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:58:34.816305 4756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:46565
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=33975
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:34.817371 4756 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:34.818706 4756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:34.828560 4762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:34.828969 4763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:36.085660 4764 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1252 milliseconds
W20250815 07:58:36.086649 4765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:36.088626 4756 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.260s user 0.000s sys 0.008s
W20250815 07:58:36.088949 4756 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.261s user 0.000s sys 0.008s
I20250815 07:58:36.089211 4756 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:36.090618 4756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:36.093647 4756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:36.095098 4756 hybrid_clock.cc:648] HybridClock initialized: now 1755244716095040 us; error 48 us; skew 500 ppm
I20250815 07:58:36.096166 4756 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:36.103994 4756 webserver.cc:480] Webserver started at http://127.1.121.254:33975/ using document root <none> and password file <none>
I20250815 07:58:36.105394 4756 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:36.105671 4756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:36.116344 4756 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250815 07:58:36.121856 4772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:36.123088 4756 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250815 07:58:36.123521 4756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "70e191f458824dfdad9ad80b91f413a6"
format_stamp: "Formatted at 2025-08-15 07:58:22 on dist-test-slave-dlgj"
I20250815 07:58:36.126312 4756 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:36.218127 4756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:36.219507 4756 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:36.219848 4756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:36.282549 4756 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:46565
I20250815 07:58:36.282593 4823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:46565 every 8 connection(s)
I20250815 07:58:36.285128 4756 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:58:36.288229 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 4756
I20250815 07:58:36.289925 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:37599
--local_ip_for_outbound_sockets=127.1.121.193
--tserver_master_addrs=127.1.121.254:46565
--webserver_port=44205
--webserver_interface=127.1.121.193
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:36.298650 4824 sys_catalog.cc:263] Verifying existing consensus state
I20250815 07:58:36.303190 4824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap starting.
I20250815 07:58:36.312080 4824 log.cc:826] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:36.356374 4824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:36.357129 4824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap complete.
I20250815 07:58:36.375074 4824 raft_consensus.cc:357] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:36.376837 4824 raft_consensus.cc:738] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e191f458824dfdad9ad80b91f413a6, State: Initialized, Role: FOLLOWER
I20250815 07:58:36.377466 4824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:36.378010 4824 raft_consensus.cc:397] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:36.378257 4824 raft_consensus.cc:491] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:36.378520 4824 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 2 FOLLOWER]: Advancing to term 3
I20250815 07:58:36.383306 4824 raft_consensus.cc:513] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:36.383788 4824 leader_election.cc:304] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e191f458824dfdad9ad80b91f413a6; no voters:
I20250815 07:58:36.385638 4824 leader_election.cc:290] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250815 07:58:36.385968 4828 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 07:58:36.388862 4828 raft_consensus.cc:695] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 3 LEADER]: Becoming Leader. State: Replica: 70e191f458824dfdad9ad80b91f413a6, State: Running, Role: LEADER
I20250815 07:58:36.389595 4828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:36.389932 4824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:58:36.396317 4829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "70e191f458824dfdad9ad80b91f413a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } } }
I20250815 07:58:36.396976 4829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:36.399312 4830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e191f458824dfdad9ad80b91f413a6. Latest consensus state: current_term: 3 leader_uuid: "70e191f458824dfdad9ad80b91f413a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } } }
I20250815 07:58:36.399843 4830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:36.406773 4834 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:58:36.430528 4834 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=4ef17d00c5654caf900bfe447aec6d4f]
I20250815 07:58:36.434216 4834 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=7d15af883af14b0aab526d8d6446428f]
I20250815 07:58:36.435964 4834 catalog_manager.cc:671] Loaded metadata for table TestTable [id=a4b07683e888402498cd1ab11676bc0e]
I20250815 07:58:36.447393 4834 tablet_loader.cc:96] loaded metadata for tablet 3150c850797343f4bbab2cc804e4bc88 (table TestTable [id=a4b07683e888402498cd1ab11676bc0e])
I20250815 07:58:36.448724 4834 tablet_loader.cc:96] loaded metadata for tablet 47b2ca7263944000837d35652b69ec59 (table TestTable1 [id=7d15af883af14b0aab526d8d6446428f])
I20250815 07:58:36.450196 4834 tablet_loader.cc:96] loaded metadata for tablet cb3af857207e4386ad2828460e9a90db (table TestTable2 [id=4ef17d00c5654caf900bfe447aec6d4f])
I20250815 07:58:36.452157 4834 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:58:36.458792 4834 catalog_manager.cc:1261] Loaded cluster ID: 0927bf0b94ce495f8be57a0848bb40f8
I20250815 07:58:36.459126 4834 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:58:36.468870 4834 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:58:36.476054 4834 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Loaded TSK: 0
I20250815 07:58:36.478381 4834 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:58:36.484400 4846 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250815 07:58:36.634029 4826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:36.634445 4826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:36.634856 4826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:36.661007 4826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:36.661784 4826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:58:36.690238 4826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:37599
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=44205
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:36.691401 4826 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:36.692894 4826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:36.703508 4852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:36.704690 4853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:37.784029 4826 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.078s user 0.005s sys 0.002s
W20250815 07:58:37.784391 4826 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.079s user 0.005s sys 0.002s
W20250815 07:58:37.785058 4855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:37.784917 4826 server_base.cc:1047] running on GCE node
I20250815 07:58:37.786152 4826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:37.788518 4826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:37.789924 4826 hybrid_clock.cc:648] HybridClock initialized: now 1755244717789866 us; error 57 us; skew 500 ppm
I20250815 07:58:37.790611 4826 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:37.797525 4826 webserver.cc:480] Webserver started at http://127.1.121.193:44205/ using document root <none> and password file <none>
I20250815 07:58:37.798326 4826 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:37.798539 4826 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:37.805894 4826 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20250815 07:58:37.810163 4862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:37.811126 4826 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250815 07:58:37.811458 4826 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95"
format_stamp: "Formatted at 2025-08-15 07:58:24 on dist-test-slave-dlgj"
I20250815 07:58:37.813232 4826 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:37.874400 4826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:37.875603 4826 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:37.875962 4826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:37.878608 4826 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:37.884651 4869 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 07:58:37.891222 4826 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 07:58:37.891402 4826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.002s sys 0.000s
I20250815 07:58:37.891681 4826 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 07:58:37.895694 4826 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 07:58:37.895968 4826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.005s sys 0.000s
I20250815 07:58:37.896243 4869 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap starting.
I20250815 07:58:37.968071 4869 log.cc:826] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:38.071190 4826 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:37599
I20250815 07:58:38.071452 4976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:37599 every 8 connection(s)
I20250815 07:58:38.074301 4826 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:58:38.076773 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 4826
I20250815 07:58:38.078253 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:37329
--local_ip_for_outbound_sockets=127.1.121.194
--tserver_master_addrs=127.1.121.254:46565
--webserver_port=42825
--webserver_interface=127.1.121.194
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:38.125150 4977 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:38.125566 4977 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:38.126575 4977 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:38.127228 4869 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:38.128074 4869 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap complete.
I20250815 07:58:38.129499 4869 ts_tablet_manager.cc:1397] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent bootstrapping tablet: real 0.234s user 0.181s sys 0.039s
I20250815 07:58:38.130750 4789 ts_manager.cc:194] Registered new tserver with Master: 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193:37599)
I20250815 07:58:38.137665 4789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:37681
I20250815 07:58:38.144255 4869 raft_consensus.cc:357] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:38.146935 4869 raft_consensus.cc:738] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Initialized, Role: FOLLOWER
I20250815 07:58:38.147627 4869 consensus_queue.cc:260] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:38.148182 4869 raft_consensus.cc:397] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:38.148456 4869 raft_consensus.cc:491] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:38.149195 4869 raft_consensus.cc:3058] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 1 FOLLOWER]: Advancing to term 2
I20250815 07:58:38.156713 4869 raft_consensus.cc:513] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:38.157418 4869 leader_election.cc:304] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cb7f777e88841a3a4aa5b37a1ab2b95; no voters:
I20250815 07:58:38.159387 4869 leader_election.cc:290] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 07:58:38.159830 4982 raft_consensus.cc:2802] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 07:58:38.167703 4982 raft_consensus.cc:695] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEADER]: Becoming Leader. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Running, Role: LEADER
I20250815 07:58:38.168251 4869 ts_tablet_manager.cc:1428] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent starting tablet: real 0.039s user 0.031s sys 0.008s
I20250815 07:58:38.168859 4977 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
I20250815 07:58:38.168566 4982 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:38.178200 4789 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: term changed from 0 to 2, leader changed from <none> to 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193), VOTER 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193) added. New cstate: current_term: 2 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } }
W20250815 07:58:38.218823 4789 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3150c850797343f4bbab2cc804e4bc88 with cas_config_opid_index -1: no extra replica candidate found for tablet 3150c850797343f4bbab2cc804e4bc88 (table TestTable [id=a4b07683e888402498cd1ab11676bc0e]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250815 07:58:38.389703 4978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:38.390134 4978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:38.390558 4978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:38.417392 4978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:38.418102 4978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:58:38.447857 4978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:37329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=42825
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:38.449003 4978 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:38.450376 4978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:38.461115 4996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:39.864230 4995 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4978
W20250815 07:58:40.026878 4978 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.566s user 0.000s sys 0.001s
W20250815 07:58:38.462466 4997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:40.027262 4978 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.566s user 0.000s sys 0.002s
W20250815 07:58:40.027560 4998 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1565 milliseconds
I20250815 07:58:40.028868 4978 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 07:58:40.028923 4999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:40.031527 4978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:40.033396 4978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:40.034699 4978 hybrid_clock.cc:648] HybridClock initialized: now 1755244720034663 us; error 39 us; skew 500 ppm
I20250815 07:58:40.035439 4978 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:40.040596 4978 webserver.cc:480] Webserver started at http://127.1.121.194:42825/ using document root <none> and password file <none>
I20250815 07:58:40.041409 4978 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:40.041608 4978 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:40.048308 4978 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20250815 07:58:40.052280 5007 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:40.053143 4978 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250815 07:58:40.053400 4978 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "eb9b068c8b91482fb5236dccf294f0d4"
format_stamp: "Formatted at 2025-08-15 07:58:27 on dist-test-slave-dlgj"
I20250815 07:58:40.055002 4978 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:40.103956 4978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:40.105298 4978 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:40.105679 4978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:40.107887 4978 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:40.112993 5014 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 07:58:40.119267 4978 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 07:58:40.119451 4978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.002s
I20250815 07:58:40.119663 4978 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 07:58:40.123615 4978 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 07:58:40.123771 4978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.001s
I20250815 07:58:40.124076 5014 tablet_bootstrap.cc:492] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap starting.
I20250815 07:58:40.185551 5014 log.cc:826] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:40.267983 5014 tablet_bootstrap.cc:492] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:40.268723 5014 tablet_bootstrap.cc:492] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap complete.
I20250815 07:58:40.269851 5014 ts_tablet_manager.cc:1397] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Time spent bootstrapping tablet: real 0.146s user 0.114s sys 0.029s
I20250815 07:58:40.284941 5014 raft_consensus.cc:357] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:40.287554 5014 raft_consensus.cc:738] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Initialized, Role: FOLLOWER
I20250815 07:58:40.288326 5014 consensus_queue.cc:260] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:40.288841 5014 raft_consensus.cc:397] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:40.289083 5014 raft_consensus.cc:491] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:40.289382 5014 raft_consensus.cc:3058] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 1 FOLLOWER]: Advancing to term 2
I20250815 07:58:40.296325 5014 raft_consensus.cc:513] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:40.297158 5014 leader_election.cc:304] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9b068c8b91482fb5236dccf294f0d4; no voters:
I20250815 07:58:40.299572 4978 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:37329
I20250815 07:58:40.299671 5121 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:37329 every 8 connection(s)
I20250815 07:58:40.300470 5014 leader_election.cc:290] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 07:58:40.300760 5122 raft_consensus.cc:2802] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 07:58:40.302758 5122 raft_consensus.cc:695] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEADER]: Becoming Leader. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Running, Role: LEADER
I20250815 07:58:40.303084 4978 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 07:58:40.303440 5122 consensus_queue.cc:237] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:40.309856 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 4978
I20250815 07:58:40.310760 5014 ts_tablet_manager.cc:1428] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Time spent starting tablet: real 0.041s user 0.035s sys 0.000s
I20250815 07:58:40.312047 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:43941
--local_ip_for_outbound_sockets=127.1.121.195
--tserver_master_addrs=127.1.121.254:46565
--webserver_port=40185
--webserver_interface=127.1.121.195
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:40.328804 5125 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:40.329161 5125 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:40.329963 5125 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:40.333099 4789 ts_manager.cc:194] Registered new tserver with Master: eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329)
I20250815 07:58:40.334440 4789 catalog_manager.cc:5582] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "eb9b068c8b91482fb5236dccf294f0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } health_report { overall_health: HEALTHY } } }
I20250815 07:58:40.344115 4789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:53483
I20250815 07:58:40.346643 5125 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
I20250815 07:58:40.530884 4932 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } }
I20250815 07:58:40.536087 5139 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index -1 to 11, NON_VOTER eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } } }
I20250815 07:58:40.546469 4776 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3150c850797343f4bbab2cc804e4bc88 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250815 07:58:40.549984 4789 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: config changed from index -1 to 11, NON_VOTER eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) added. New cstate: current_term: 2 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250815 07:58:40.559456 4864 consensus_peers.cc:489] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 -> Peer eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329): Couldn't send request to peer eb9b068c8b91482fb5236dccf294f0d4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3150c850797343f4bbab2cc804e4bc88. This is attempt 1: this message will repeat every 5th retry.
W20250815 07:58:40.597843 5131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:40.598275 5131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:40.598706 5131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:40.625897 5131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:40.626643 5131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:58:40.657909 5131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:43941
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=40185
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:40.659222 5131 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:40.660740 5131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:40.670516 5148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:40.944648 5154 ts_tablet_manager.cc:927] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Initiating tablet copy from peer 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193:37599)
I20250815 07:58:40.946786 5154 tablet_copy_client.cc:323] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: tablet copy: Beginning tablet copy session from remote peer at address 127.1.121.193:37599
I20250815 07:58:40.958611 4952 tablet_copy_service.cc:140] P 2cb7f777e88841a3a4aa5b37a1ab2b95: Received BeginTabletCopySession request for tablet 3150c850797343f4bbab2cc804e4bc88 from peer eb9b068c8b91482fb5236dccf294f0d4 ({username='slave'} at 127.1.121.194:59151)
I20250815 07:58:40.959365 4952 tablet_copy_service.cc:161] P 2cb7f777e88841a3a4aa5b37a1ab2b95: Beginning new tablet copy session on tablet 3150c850797343f4bbab2cc804e4bc88 from peer eb9b068c8b91482fb5236dccf294f0d4 at {username='slave'} at 127.1.121.194:59151: session id = eb9b068c8b91482fb5236dccf294f0d4-3150c850797343f4bbab2cc804e4bc88
I20250815 07:58:40.969306 4952 tablet_copy_source_session.cc:215] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 07:58:40.973652 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3150c850797343f4bbab2cc804e4bc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:40.984236 5154 tablet_copy_client.cc:806] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: tablet copy: Starting download of 0 data blocks...
I20250815 07:58:40.984716 5154 tablet_copy_client.cc:670] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: tablet copy: Starting download of 1 WAL segments...
I20250815 07:58:40.989817 5154 tablet_copy_client.cc:538] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 07:58:40.994575 5154 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap starting.
I20250815 07:58:41.076617 5154 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:41.077121 5154 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap complete.
I20250815 07:58:41.077498 5154 ts_tablet_manager.cc:1397] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Time spent bootstrapping tablet: real 0.083s user 0.064s sys 0.020s
I20250815 07:58:41.078823 5154 raft_consensus.cc:357] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } }
I20250815 07:58:41.079182 5154 raft_consensus.cc:738] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Initialized, Role: LEARNER
I20250815 07:58:41.079547 5154 consensus_queue.cc:260] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } }
I20250815 07:58:41.082062 5154 ts_tablet_manager.cc:1428] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250815 07:58:41.083694 4952 tablet_copy_service.cc:342] P 2cb7f777e88841a3a4aa5b37a1ab2b95: Request end of tablet copy session eb9b068c8b91482fb5236dccf294f0d4-3150c850797343f4bbab2cc804e4bc88 received from {username='slave'} at 127.1.121.194:59151
I20250815 07:58:41.084154 4952 tablet_copy_service.cc:434] P 2cb7f777e88841a3a4aa5b37a1ab2b95: ending tablet copy session eb9b068c8b91482fb5236dccf294f0d4-3150c850797343f4bbab2cc804e4bc88 on tablet 3150c850797343f4bbab2cc804e4bc88 with peer eb9b068c8b91482fb5236dccf294f0d4
I20250815 07:58:41.449324 5077 raft_consensus.cc:1215] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20250815 07:58:40.674994 5151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:40.671913 5149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:41.713639 5150 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 07:58:41.713687 5131 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:41.716830 5131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:41.718654 5131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:41.719952 5131 hybrid_clock.cc:648] HybridClock initialized: now 1755244721719925 us; error 31 us; skew 500 ppm
I20250815 07:58:41.720774 5131 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:41.726759 5131 webserver.cc:480] Webserver started at http://127.1.121.195:40185/ using document root <none> and password file <none>
I20250815 07:58:41.727530 5131 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:41.727732 5131 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:41.734223 5131 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250815 07:58:41.737993 5166 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:41.738806 5131 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250815 07:58:41.739073 5131 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee"
format_stamp: "Formatted at 2025-08-15 07:58:29 on dist-test-slave-dlgj"
I20250815 07:58:41.740790 5131 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:41.803194 5131 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:41.804463 5131 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:41.804804 5131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:41.806855 5131 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:41.811486 5173 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 07:58:41.817899 5131 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 07:58:41.818132 5131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20250815 07:58:41.818404 5131 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 07:58:41.822573 5131 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 07:58:41.822727 5131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.002s
I20250815 07:58:41.823061 5173 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap starting.
I20250815 07:58:41.869864 5173 log.cc:826] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:41.944959 5161 raft_consensus.cc:1062] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: attempting to promote NON_VOTER eb9b068c8b91482fb5236dccf294f0d4 to VOTER
I20250815 07:58:41.946802 5161 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:41.952756 5077 raft_consensus.cc:1273] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEARNER]: Refusing update from remote peer 2cb7f777e88841a3a4aa5b37a1ab2b95: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 07:58:41.954533 5160 consensus_queue.cc:1035] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250815 07:58:41.963268 5160 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:41.964612 5077 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:41.973289 4788 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: config changed from index 11 to 12, eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 07:58:41.990201 5173 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:41.990840 5131 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:43941
I20250815 07:58:41.990944 5286 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:43941 every 8 connection(s)
I20250815 07:58:41.991164 5173 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap complete.
I20250815 07:58:41.992995 5173 ts_tablet_manager.cc:1397] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent bootstrapping tablet: real 0.170s user 0.137s sys 0.028s
I20250815 07:58:41.993969 5131 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 07:58:41.997572 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 5131
I20250815 07:58:42.017621 5173 raft_consensus.cc:357] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } }
I20250815 07:58:42.020961 5173 raft_consensus.cc:738] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1eea6c51dc4358a6e5cddd3cb203ee, State: Initialized, Role: FOLLOWER
I20250815 07:58:42.021914 5173 consensus_queue.cc:260] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } }
I20250815 07:58:42.022761 5173 raft_consensus.cc:397] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:42.023168 5173 raft_consensus.cc:491] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:42.023622 5173 raft_consensus.cc:3058] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 1 FOLLOWER]: Advancing to term 2
I20250815 07:58:42.032845 5173 raft_consensus.cc:513] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } }
I20250815 07:58:42.033615 5173 leader_election.cc:304] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1eea6c51dc4358a6e5cddd3cb203ee; no voters:
I20250815 07:58:42.037164 5173 leader_election.cc:290] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 07:58:42.039932 5294 raft_consensus.cc:2802] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Leader election won for term 2
I20250815 07:58:42.041121 5173 ts_tablet_manager.cc:1428] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent starting tablet: real 0.048s user 0.043s sys 0.001s
I20250815 07:58:42.041870 5294 raft_consensus.cc:695] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEADER]: Becoming Leader. State: Replica: bd1eea6c51dc4358a6e5cddd3cb203ee, State: Running, Role: LEADER
I20250815 07:58:42.042508 5294 consensus_queue.cc:237] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } }
I20250815 07:58:42.046802 5288 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:42.047222 5288 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:42.048188 5288 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:42.051749 4788 ts_manager.cc:194] Registered new tserver with Master: bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941)
I20250815 07:58:42.052678 4788 catalog_manager.cc:5582] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee reported cstate change: term changed from 0 to 2, leader changed from <none> to bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195), VOTER bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) added. New cstate: current_term: 2 leader_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } health_report { overall_health: HEALTHY } } }
I20250815 07:58:42.060680 4788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:38139
I20250815 07:58:42.063854 5288 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
I20250815 07:58:42.064363 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:58:42.069041 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250815 07:58:42.072748 1511 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250815 07:58:42.075616 5236 consensus_queue.cc:237] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } }
I20250815 07:58:42.078286 5294 raft_consensus.cc:2953] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } }
I20250815 07:58:42.085343 4776 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 47b2ca7263944000837d35652b69ec59 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250815 07:58:42.088807 5170 consensus_peers.cc:489] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee -> Peer 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193:37599): Couldn't send request to peer 2cb7f777e88841a3a4aa5b37a1ab2b95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47b2ca7263944000837d35652b69ec59. This is attempt 1: this message will repeat every 5th retry.
I20250815 07:58:42.088460 4788 catalog_manager.cc:5582] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee reported cstate change: config changed from index -1 to 9, NON_VOTER 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193) added. New cstate: current_term: 2 leader_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 07:58:42.097620 5236 consensus_queue.cc:237] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } }
I20250815 07:58:42.100350 5296 raft_consensus.cc:2953] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } } }
W20250815 07:58:42.101692 5170 consensus_peers.cc:489] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee -> Peer 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193:37599): Couldn't send request to peer 2cb7f777e88841a3a4aa5b37a1ab2b95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47b2ca7263944000837d35652b69ec59. This is attempt 1: this message will repeat every 5th retry.
I20250815 07:58:42.106281 4776 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 47b2ca7263944000837d35652b69ec59 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250815 07:58:42.108450 4788 catalog_manager.cc:5582] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee reported cstate change: config changed from index 9 to 10, NON_VOTER eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) added. New cstate: current_term: 2 leader_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250815 07:58:42.109493 5168 consensus_peers.cc:489] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee -> Peer eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329): Couldn't send request to peer eb9b068c8b91482fb5236dccf294f0d4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47b2ca7263944000837d35652b69ec59. This is attempt 1: this message will repeat every 5th retry.
I20250815 07:58:42.109934 4932 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: NON_VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: true } }
I20250815 07:58:42.114073 5077 raft_consensus.cc:1273] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Refusing update from remote peer 2cb7f777e88841a3a4aa5b37a1ab2b95: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250815 07:58:42.115118 5161 consensus_queue.cc:1035] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250815 07:58:42.120564 5160 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, NON_VOTER bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) added. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: NON_VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: true } } }
I20250815 07:58:42.121994 5077 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, NON_VOTER bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) added. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: NON_VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: true } } }
W20250815 07:58:42.124512 4866 consensus_peers.cc:489] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 -> Peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Couldn't send request to peer bd1eea6c51dc4358a6e5cddd3cb203ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3150c850797343f4bbab2cc804e4bc88. This is attempt 1: this message will repeat every 5th retry.
I20250815 07:58:42.128858 4776 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3150c850797343f4bbab2cc804e4bc88 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20250815 07:58:42.131256 4788 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: config changed from index 12 to 13, NON_VOTER bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) added. New cstate: current_term: 2 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: NON_VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250815 07:58:42.545375 5304 ts_tablet_manager.cc:927] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Initiating tablet copy from peer 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193:37599)
I20250815 07:58:42.547473 5304 tablet_copy_client.cc:323] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: tablet copy: Beginning tablet copy session from remote peer at address 127.1.121.193:37599
I20250815 07:58:42.548741 4952 tablet_copy_service.cc:140] P 2cb7f777e88841a3a4aa5b37a1ab2b95: Received BeginTabletCopySession request for tablet 3150c850797343f4bbab2cc804e4bc88 from peer bd1eea6c51dc4358a6e5cddd3cb203ee ({username='slave'} at 127.1.121.195:33981)
I20250815 07:58:42.549086 4952 tablet_copy_service.cc:161] P 2cb7f777e88841a3a4aa5b37a1ab2b95: Beginning new tablet copy session on tablet 3150c850797343f4bbab2cc804e4bc88 from peer bd1eea6c51dc4358a6e5cddd3cb203ee at {username='slave'} at 127.1.121.195:33981: session id = bd1eea6c51dc4358a6e5cddd3cb203ee-3150c850797343f4bbab2cc804e4bc88
I20250815 07:58:42.552794 4952 tablet_copy_source_session.cc:215] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 07:58:42.555338 5304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3150c850797343f4bbab2cc804e4bc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:42.555693 5306 ts_tablet_manager.cc:927] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Initiating tablet copy from peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941)
I20250815 07:58:42.557097 5306 tablet_copy_client.cc:323] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: tablet copy: Beginning tablet copy session from remote peer at address 127.1.121.195:43941
I20250815 07:58:42.566980 5256 tablet_copy_service.cc:140] P bd1eea6c51dc4358a6e5cddd3cb203ee: Received BeginTabletCopySession request for tablet 47b2ca7263944000837d35652b69ec59 from peer eb9b068c8b91482fb5236dccf294f0d4 ({username='slave'} at 127.1.121.194:60827)
I20250815 07:58:42.567525 5256 tablet_copy_service.cc:161] P bd1eea6c51dc4358a6e5cddd3cb203ee: Beginning new tablet copy session on tablet 47b2ca7263944000837d35652b69ec59 from peer eb9b068c8b91482fb5236dccf294f0d4 at {username='slave'} at 127.1.121.194:60827: session id = eb9b068c8b91482fb5236dccf294f0d4-47b2ca7263944000837d35652b69ec59
I20250815 07:58:42.570461 5304 tablet_copy_client.cc:806] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: tablet copy: Starting download of 0 data blocks...
I20250815 07:58:42.570999 5304 tablet_copy_client.cc:670] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: tablet copy: Starting download of 1 WAL segments...
I20250815 07:58:42.573287 5256 tablet_copy_source_session.cc:215] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 07:58:42.574944 5304 tablet_copy_client.cc:538] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 07:58:42.575902 5306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b2ca7263944000837d35652b69ec59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:42.579703 5304 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap starting.
I20250815 07:58:42.584621 5306 tablet_copy_client.cc:806] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: tablet copy: Starting download of 0 data blocks...
I20250815 07:58:42.585033 5306 tablet_copy_client.cc:670] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: tablet copy: Starting download of 1 WAL segments...
I20250815 07:58:42.588662 5306 tablet_copy_client.cc:538] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 07:58:42.593209 5306 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap starting.
I20250815 07:58:42.634951 5311 ts_tablet_manager.cc:927] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Initiating tablet copy from peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941)
I20250815 07:58:42.637276 5311 tablet_copy_client.cc:323] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: tablet copy: Beginning tablet copy session from remote peer at address 127.1.121.195:43941
I20250815 07:58:42.638890 5256 tablet_copy_service.cc:140] P bd1eea6c51dc4358a6e5cddd3cb203ee: Received BeginTabletCopySession request for tablet 47b2ca7263944000837d35652b69ec59 from peer 2cb7f777e88841a3a4aa5b37a1ab2b95 ({username='slave'} at 127.1.121.193:59857)
I20250815 07:58:42.639345 5256 tablet_copy_service.cc:161] P bd1eea6c51dc4358a6e5cddd3cb203ee: Beginning new tablet copy session on tablet 47b2ca7263944000837d35652b69ec59 from peer 2cb7f777e88841a3a4aa5b37a1ab2b95 at {username='slave'} at 127.1.121.193:59857: session id = 2cb7f777e88841a3a4aa5b37a1ab2b95-47b2ca7263944000837d35652b69ec59
I20250815 07:58:42.645493 5256 tablet_copy_source_session.cc:215] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Tablet Copy: opened 0 blocks and 1 log segments
I20250815 07:58:42.648396 5311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47b2ca7263944000837d35652b69ec59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:58:42.661605 5311 tablet_copy_client.cc:806] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: tablet copy: Starting download of 0 data blocks...
I20250815 07:58:42.662147 5311 tablet_copy_client.cc:670] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: tablet copy: Starting download of 1 WAL segments...
I20250815 07:58:42.665779 5311 tablet_copy_client.cc:538] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250815 07:58:42.672986 5311 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap starting.
I20250815 07:58:42.677865 5304 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:42.678364 5304 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap complete.
I20250815 07:58:42.678742 5304 ts_tablet_manager.cc:1397] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent bootstrapping tablet: real 0.099s user 0.091s sys 0.008s
I20250815 07:58:42.680173 5304 raft_consensus.cc:357] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: NON_VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: true } }
I20250815 07:58:42.680598 5304 raft_consensus.cc:738] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: bd1eea6c51dc4358a6e5cddd3cb203ee, State: Initialized, Role: LEARNER
I20250815 07:58:42.680982 5304 consensus_queue.cc:260] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: NON_VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: true } }
I20250815 07:58:42.682760 5304 ts_tablet_manager.cc:1428] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250815 07:58:42.684518 5306 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:42.684518 4952 tablet_copy_service.cc:342] P 2cb7f777e88841a3a4aa5b37a1ab2b95: Request end of tablet copy session bd1eea6c51dc4358a6e5cddd3cb203ee-3150c850797343f4bbab2cc804e4bc88 received from {username='slave'} at 127.1.121.195:33981
I20250815 07:58:42.684845 4952 tablet_copy_service.cc:434] P 2cb7f777e88841a3a4aa5b37a1ab2b95: ending tablet copy session bd1eea6c51dc4358a6e5cddd3cb203ee-3150c850797343f4bbab2cc804e4bc88 on tablet 3150c850797343f4bbab2cc804e4bc88 with peer bd1eea6c51dc4358a6e5cddd3cb203ee
I20250815 07:58:42.685213 5306 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap complete.
I20250815 07:58:42.685675 5306 ts_tablet_manager.cc:1397] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Time spent bootstrapping tablet: real 0.093s user 0.080s sys 0.010s
I20250815 07:58:42.687494 5306 raft_consensus.cc:357] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } }
I20250815 07:58:42.687891 5306 raft_consensus.cc:738] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Initialized, Role: LEARNER
I20250815 07:58:42.688308 5306 consensus_queue.cc:260] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } }
I20250815 07:58:42.692255 5306 ts_tablet_manager.cc:1428] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250815 07:58:42.693842 5256 tablet_copy_service.cc:342] P bd1eea6c51dc4358a6e5cddd3cb203ee: Request end of tablet copy session eb9b068c8b91482fb5236dccf294f0d4-47b2ca7263944000837d35652b69ec59 received from {username='slave'} at 127.1.121.194:60827
I20250815 07:58:42.694211 5256 tablet_copy_service.cc:434] P bd1eea6c51dc4358a6e5cddd3cb203ee: ending tablet copy session eb9b068c8b91482fb5236dccf294f0d4-47b2ca7263944000837d35652b69ec59 on tablet 47b2ca7263944000837d35652b69ec59 with peer eb9b068c8b91482fb5236dccf294f0d4
I20250815 07:58:42.749643 5311 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:42.750200 5311 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap complete.
I20250815 07:58:42.750573 5311 ts_tablet_manager.cc:1397] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent bootstrapping tablet: real 0.078s user 0.062s sys 0.012s
I20250815 07:58:42.752408 5311 raft_consensus.cc:357] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } }
I20250815 07:58:42.752815 5311 raft_consensus.cc:738] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Initialized, Role: LEARNER
I20250815 07:58:42.753234 5311 consensus_queue.cc:260] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: NON_VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: true } }
I20250815 07:58:42.754577 5311 ts_tablet_manager.cc:1428] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250815 07:58:42.755867 5256 tablet_copy_service.cc:342] P bd1eea6c51dc4358a6e5cddd3cb203ee: Request end of tablet copy session 2cb7f777e88841a3a4aa5b37a1ab2b95-47b2ca7263944000837d35652b69ec59 received from {username='slave'} at 127.1.121.193:59857
I20250815 07:58:42.756247 5256 tablet_copy_service.cc:434] P bd1eea6c51dc4358a6e5cddd3cb203ee: ending tablet copy session 2cb7f777e88841a3a4aa5b37a1ab2b95-47b2ca7263944000837d35652b69ec59 on tablet 47b2ca7263944000837d35652b69ec59 with peer 2cb7f777e88841a3a4aa5b37a1ab2b95
I20250815 07:58:42.774931 4775 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3150c850797343f4bbab2cc804e4bc88 with cas_config_opid_index 11: aborting the task: latest config opid_index 13; task opid_index 11
I20250815 07:58:43.018682 5077 raft_consensus.cc:1215] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250815 07:58:43.076725 4932 raft_consensus.cc:1215] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250815 07:58:43.077206 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2cb7f777e88841a3a4aa5b37a1ab2b95 to finish bootstrapping
I20250815 07:58:43.096829 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver eb9b068c8b91482fb5236dccf294f0d4 to finish bootstrapping
I20250815 07:58:43.108398 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bd1eea6c51dc4358a6e5cddd3cb203ee to finish bootstrapping
I20250815 07:58:43.119508 5236 raft_consensus.cc:1215] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEARNER]: Deduplicated request from leader. Original: 2.12->[2.13-2.13] Dedup: 2.13->[]
I20250815 07:58:43.366366 4912 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 07:58:43.370573 5216 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 07:58:43.371654 5057 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 07:58:43.545944 5322 raft_consensus.cc:1062] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: attempting to promote NON_VOTER bd1eea6c51dc4358a6e5cddd3cb203ee to VOTER
I20250815 07:58:43.547634 5322 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:43.551296 5316 raft_consensus.cc:1062] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: attempting to promote NON_VOTER eb9b068c8b91482fb5236dccf294f0d4 to VOTER
I20250815 07:58:43.553128 5236 raft_consensus.cc:1273] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEARNER]: Refusing update from remote peer 2cb7f777e88841a3a4aa5b37a1ab2b95: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250815 07:58:43.553179 5316 consensus_queue.cc:237] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:43.553717 5077 raft_consensus.cc:1273] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Refusing update from remote peer 2cb7f777e88841a3a4aa5b37a1ab2b95: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250815 07:58:43.554687 5161 consensus_queue.cc:1035] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250815 07:58:43.555480 5321 consensus_queue.cc:1035] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
70e191f458824dfdad9ad80b91f413a6 | 127.1.121.254:46565 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.1.121.212:40013 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2cb7f777e88841a3a4aa5b37a1ab2b95 | 127.1.121.193:37599 | HEALTHY | <none> | 1 | 0
bd1eea6c51dc4358a6e5cddd3cb203ee | 127.1.121.195:43941 | HEALTHY | <none> | 1 | 0
eb9b068c8b91482fb5236dccf294f0d4 | 127.1.121.194:37329 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.121.193 | experimental | 127.1.121.193:37599
local_ip_for_outbound_sockets | 127.1.121.194 | experimental | 127.1.121.194:37329
local_ip_for_outbound_sockets | 127.1.121.195 | experimental | 127.1.121.195:43941
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.121.193:37599
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.121.194:37329
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.121.195:43941
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.1.121.212:40013 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
I20250815 07:58:43.573990 5321 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } } }
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 07:58:43.577365 5077 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } } }
I20250815 07:58:43.588214 4787 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: config changed from index 13 to 14, bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250815 07:58:43.589094 5076 raft_consensus.cc:1273] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 LEARNER]: Refusing update from remote peer bd1eea6c51dc4358a6e5cddd3cb203ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 07:58:43.591120 4932 raft_consensus.cc:1273] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEARNER]: Refusing update from remote peer bd1eea6c51dc4358a6e5cddd3cb203ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250815 07:58:43.591957 5316 consensus_queue.cc:1035] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250815 07:58:43.592867 5294 consensus_queue.cc:1035] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250815 07:58:43.593472 5236 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } } }
I20250815 07:58:43.595485 1511 log_verifier.cc:126] Checking tablet 3150c850797343f4bbab2cc804e4bc88
I20250815 07:58:43.602213 5294 raft_consensus.cc:1025] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEADER]: attempt to promote peer 2cb7f777e88841a3a4aa5b37a1ab2b95: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250815 07:58:43.609489 5315 raft_consensus.cc:2953] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:43.610284 5076 raft_consensus.cc:2953] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:43.621014 4932 raft_consensus.cc:2953] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEARNER]: Committing config change with OpId 2.11: config changed from index 10 to 11, eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:43.632177 4788 catalog_manager.cc:5582] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 reported cstate change: config changed from index 10 to 11, eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: NON_VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: true } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:43.633172 5315 raft_consensus.cc:1062] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: attempting to promote NON_VOTER 2cb7f777e88841a3a4aa5b37a1ab2b95 to VOTER
I20250815 07:58:43.634807 5315 consensus_queue.cc:237] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:43.639910 5076 raft_consensus.cc:1273] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Refusing update from remote peer bd1eea6c51dc4358a6e5cddd3cb203ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 07:58:43.641029 5315 consensus_queue.cc:1035] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250815 07:58:43.641419 4932 raft_consensus.cc:1273] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 LEARNER]: Refusing update from remote peer bd1eea6c51dc4358a6e5cddd3cb203ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250815 07:58:43.642969 5315 consensus_queue.cc:1035] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250815 07:58:43.646423 5316 raft_consensus.cc:2953] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:43.648608 5076 raft_consensus.cc:2953] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:43.654021 4931 raft_consensus.cc:2953] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:43.657207 4788 catalog_manager.cc:5582] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee reported cstate change: config changed from index 11 to 12, 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250815 07:58:43.709107 1511 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 3150c850797343f4bbab2cc804e4bc88
I20250815 07:58:43.709367 1511 log_verifier.cc:126] Checking tablet 47b2ca7263944000837d35652b69ec59
I20250815 07:58:43.778656 1511 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 47b2ca7263944000837d35652b69ec59
I20250815 07:58:43.778882 1511 log_verifier.cc:126] Checking tablet cb3af857207e4386ad2828460e9a90db
I20250815 07:58:43.800696 1511 log_verifier.cc:177] Verified matching terms for 8 ops in tablet cb3af857207e4386ad2828460e9a90db
I20250815 07:58:43.800992 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 4756
I20250815 07:58:43.827704 1511 minidump.cc:252] Setting minidump size limit to 20M
I20250815 07:58:43.828810 1511 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:43.829655 1511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:43.838718 5360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:43.838997 5361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:44.019637 5363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:44.019706 1511 server_base.cc:1047] running on GCE node
I20250815 07:58:44.021054 1511 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250815 07:58:44.021332 1511 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250815 07:58:44.021512 1511 hybrid_clock.cc:648] HybridClock initialized: now 1755244724021490 us; error 0 us; skew 500 ppm
I20250815 07:58:44.022354 1511 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:44.025575 1511 webserver.cc:480] Webserver started at http://0.0.0.0:43739/ using document root <none> and password file <none>
I20250815 07:58:44.026475 1511 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:44.026681 1511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:44.032626 1511 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250815 07:58:44.035811 5368 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:44.036670 1511 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250815 07:58:44.036908 1511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "70e191f458824dfdad9ad80b91f413a6"
format_stamp: "Formatted at 2025-08-15 07:58:22 on dist-test-slave-dlgj"
I20250815 07:58:44.038264 1511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:44.066423 1511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:44.067543 1511 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:44.067875 1511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:44.078107 1511 sys_catalog.cc:263] Verifying existing consensus state
W20250815 07:58:44.083037 1511 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.1.121.254:46565 exists but no master address supplied!
I20250815 07:58:44.085044 1511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap starting.
I20250815 07:58:44.121778 1511 log.cc:826] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:44.179370 1511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:44.180111 1511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap complete.
I20250815 07:58:44.191610 1511 raft_consensus.cc:357] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:44.192080 1511 raft_consensus.cc:738] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e191f458824dfdad9ad80b91f413a6, State: Initialized, Role: FOLLOWER
I20250815 07:58:44.192693 1511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:44.193099 1511 raft_consensus.cc:397] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:44.193297 1511 raft_consensus.cc:491] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:44.193539 1511 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 3 FOLLOWER]: Advancing to term 4
I20250815 07:58:44.197925 1511 raft_consensus.cc:513] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:44.198462 1511 leader_election.cc:304] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e191f458824dfdad9ad80b91f413a6; no voters:
I20250815 07:58:44.199615 1511 leader_election.cc:290] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250815 07:58:44.199836 5375 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 4 FOLLOWER]: Leader election won for term 4
I20250815 07:58:44.201191 5375 raft_consensus.cc:695] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 4 LEADER]: Becoming Leader. State: Replica: 70e191f458824dfdad9ad80b91f413a6, State: Running, Role: LEADER
I20250815 07:58:44.201897 5375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:44.207876 5376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "70e191f458824dfdad9ad80b91f413a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } } }
I20250815 07:58:44.208334 5376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:44.208786 5377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e191f458824dfdad9ad80b91f413a6. Latest consensus state: current_term: 4 leader_uuid: "70e191f458824dfdad9ad80b91f413a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } } }
I20250815 07:58:44.209085 5377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:44.230432 1511 tablet_replica.cc:331] stopping tablet replica
I20250815 07:58:44.230849 1511 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 4 LEADER]: Raft consensus shutting down.
I20250815 07:58:44.231199 1511 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250815 07:58:44.233131 1511 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250815 07:58:44.233563 1511 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250815 07:58:44.285722 1511 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250815 07:58:44.681903 5288 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:46565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:46565: connect: Connection refused (error 111)
W20250815 07:58:44.685948 4977 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:46565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:46565: connect: Connection refused (error 111)
W20250815 07:58:44.686949 5125 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:46565 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:46565: connect: Connection refused (error 111)
W20250815 07:58:45.303658 5118 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3180 after lost signal to thread 4991
W20250815 07:58:45.304695 5118 debug-util.cc:398] Leaking SignalData structure 0x7b0800071e00 after lost signal to thread 5121
I20250815 07:58:49.163349 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 4826
I20250815 07:58:49.187613 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 4978
I20250815 07:58:49.211484 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 5131
I20250815 07:58:49.237756 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46565
--webserver_interface=127.1.121.254
--webserver_port=33975
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:46565 with env {}
W20250815 07:58:49.507012 5448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:49.507524 5448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:49.507918 5448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:49.534812 5448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:58:49.535089 5448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:49.535320 5448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:58:49.535543 5448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:58:49.568724 5448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:46565
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=33975
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:49.569940 5448 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:49.571310 5448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:49.580411 5455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:49.581246 5456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:50.622586 5457 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1039 milliseconds
W20250815 07:58:50.623520 5459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:50.625665 5448 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.043s user 0.002s sys 0.003s
W20250815 07:58:50.625905 5448 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.044s user 0.002s sys 0.003s
I20250815 07:58:50.626092 5448 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:50.627017 5448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:50.629464 5448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:50.631052 5448 hybrid_clock.cc:648] HybridClock initialized: now 1755244730631004 us; error 58 us; skew 500 ppm
I20250815 07:58:50.631783 5448 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:50.638653 5448 webserver.cc:480] Webserver started at http://127.1.121.254:33975/ using document root <none> and password file <none>
I20250815 07:58:50.639490 5448 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:50.639676 5448 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:50.646919 5448 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250815 07:58:50.652689 5465 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:50.653699 5448 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 07:58:50.653981 5448 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "70e191f458824dfdad9ad80b91f413a6"
format_stamp: "Formatted at 2025-08-15 07:58:22 on dist-test-slave-dlgj"
I20250815 07:58:50.655596 5448 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:50.714524 5448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:50.715757 5448 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:50.716135 5448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:50.776381 5448 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:46565
I20250815 07:58:50.776430 5516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:46565 every 8 connection(s)
I20250815 07:58:50.778793 5448 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:58:50.780953 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 5448
I20250815 07:58:50.782774 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:37599
--local_ip_for_outbound_sockets=127.1.121.193
--tserver_master_addrs=127.1.121.254:46565
--webserver_port=44205
--webserver_interface=127.1.121.193
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:50.787876 5517 sys_catalog.cc:263] Verifying existing consensus state
I20250815 07:58:50.791775 5517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap starting.
I20250815 07:58:50.800462 5517 log.cc:826] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:50.866978 5517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:50.867643 5517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Bootstrap complete.
I20250815 07:58:50.883773 5517 raft_consensus.cc:357] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:50.885552 5517 raft_consensus.cc:738] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e191f458824dfdad9ad80b91f413a6, State: Initialized, Role: FOLLOWER
I20250815 07:58:50.886171 5517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:50.886592 5517 raft_consensus.cc:397] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:50.886802 5517 raft_consensus.cc:491] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:50.887035 5517 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 5 FOLLOWER]: Advancing to term 6
I20250815 07:58:50.891340 5517 raft_consensus.cc:513] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:50.891839 5517 leader_election.cc:304] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e191f458824dfdad9ad80b91f413a6; no voters:
I20250815 07:58:50.893898 5517 leader_election.cc:290] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250815 07:58:50.894250 5521 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 6 FOLLOWER]: Leader election won for term 6
I20250815 07:58:50.896996 5521 raft_consensus.cc:695] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [term 6 LEADER]: Becoming Leader. State: Replica: 70e191f458824dfdad9ad80b91f413a6, State: Running, Role: LEADER
I20250815 07:58:50.897715 5521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } }
I20250815 07:58:50.898175 5517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:58:50.904863 5523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e191f458824dfdad9ad80b91f413a6. Latest consensus state: current_term: 6 leader_uuid: "70e191f458824dfdad9ad80b91f413a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } } }
I20250815 07:58:50.904842 5522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "70e191f458824dfdad9ad80b91f413a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e191f458824dfdad9ad80b91f413a6" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46565 } } }
I20250815 07:58:50.905514 5523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:50.905618 5522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6 [sys.catalog]: This master's current role is: LEADER
I20250815 07:58:50.915503 5527 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:58:50.927119 5527 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=4ef17d00c5654caf900bfe447aec6d4f]
I20250815 07:58:50.928555 5527 catalog_manager.cc:671] Loaded metadata for table TestTable [id=79af04ffebc249b6a8f733e9cdf514ef]
I20250815 07:58:50.930248 5527 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=7d15af883af14b0aab526d8d6446428f]
I20250815 07:58:50.937021 5527 tablet_loader.cc:96] loaded metadata for tablet 3150c850797343f4bbab2cc804e4bc88 (table TestTable [id=79af04ffebc249b6a8f733e9cdf514ef])
I20250815 07:58:50.938251 5527 tablet_loader.cc:96] loaded metadata for tablet 47b2ca7263944000837d35652b69ec59 (table TestTable1 [id=7d15af883af14b0aab526d8d6446428f])
I20250815 07:58:50.939337 5527 tablet_loader.cc:96] loaded metadata for tablet cb3af857207e4386ad2828460e9a90db (table TestTable2 [id=4ef17d00c5654caf900bfe447aec6d4f])
I20250815 07:58:50.940471 5527 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:58:50.945995 5527 catalog_manager.cc:1261] Loaded cluster ID: 0927bf0b94ce495f8be57a0848bb40f8
I20250815 07:58:50.946252 5527 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:58:50.954258 5527 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:58:50.958966 5527 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 70e191f458824dfdad9ad80b91f413a6: Loaded TSK: 0
I20250815 07:58:50.960300 5527 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 07:58:51.078763 5519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:51.079195 5519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:51.079658 5519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:51.107167 5519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:51.107915 5519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:58:51.138250 5519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:37599
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=44205
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:51.139407 5519 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:51.140882 5519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:51.152199 5544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:51.153894 5545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:52.406522 5546 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250815 07:58:52.408061 5547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:52.416196 5519 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.263s user 0.439s sys 0.817s
W20250815 07:58:52.416469 5519 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.263s user 0.439s sys 0.817s
I20250815 07:58:52.416689 5519 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:52.417871 5519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:52.420233 5519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:52.421669 5519 hybrid_clock.cc:648] HybridClock initialized: now 1755244732421601 us; error 66 us; skew 500 ppm
I20250815 07:58:52.422708 5519 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:52.430768 5519 webserver.cc:480] Webserver started at http://127.1.121.193:44205/ using document root <none> and password file <none>
I20250815 07:58:52.431828 5519 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:52.432085 5519 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:52.441879 5519 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20250815 07:58:52.447470 5554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:52.448642 5519 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 07:58:52.448963 5519 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95"
format_stamp: "Formatted at 2025-08-15 07:58:24 on dist-test-slave-dlgj"
I20250815 07:58:52.451279 5519 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:52.525102 5519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:52.526896 5519 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:52.527344 5519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:52.530195 5519 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:52.536597 5561 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250815 07:58:52.546993 5519 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250815 07:58:52.547257 5519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.000s
I20250815 07:58:52.547552 5519 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250815 07:58:52.554477 5561 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap starting.
I20250815 07:58:52.558442 5519 ts_tablet_manager.cc:610] Registered 2 tablets
I20250815 07:58:52.558712 5519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.009s sys 0.000s
I20250815 07:58:52.610188 5561 log.cc:826] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:52.731118 5561 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:52.732084 5561 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap complete.
I20250815 07:58:52.733675 5561 ts_tablet_manager.cc:1397] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent bootstrapping tablet: real 0.180s user 0.135s sys 0.039s
I20250815 07:58:52.742547 5519 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:37599
I20250815 07:58:52.742645 5668 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:37599 every 8 connection(s)
I20250815 07:58:52.745050 5519 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:58:52.749040 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 5519
I20250815 07:58:52.751030 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:37329
--local_ip_for_outbound_sockets=127.1.121.194
--tserver_master_addrs=127.1.121.254:46565
--webserver_port=42825
--webserver_interface=127.1.121.194
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:52.752727 5561 raft_consensus.cc:357] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:52.755403 5561 raft_consensus.cc:738] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Initialized, Role: FOLLOWER
I20250815 07:58:52.756237 5561 consensus_queue.cc:260] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:52.798871 5561 ts_tablet_manager.cc:1428] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent starting tablet: real 0.065s user 0.028s sys 0.035s
I20250815 07:58:52.799546 5561 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap starting.
I20250815 07:58:52.814105 5669 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:52.814489 5669 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:52.815299 5669 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:52.819540 5482 ts_manager.cc:194] Registered new tserver with Master: 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193:37599)
I20250815 07:58:52.823526 5482 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: config changed from index -1 to 14, term changed from 0 to 2, VOTER 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193) added, VOTER bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) added, VOTER eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) added. New cstate: current_term: 2 committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } } }
I20250815 07:58:52.897444 5482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:36849
I20250815 07:58:52.901948 5669 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
I20250815 07:58:52.940157 5561 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:52.940804 5561 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Bootstrap complete.
I20250815 07:58:52.941825 5561 ts_tablet_manager.cc:1397] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent bootstrapping tablet: real 0.142s user 0.129s sys 0.010s
I20250815 07:58:52.943156 5561 raft_consensus.cc:357] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:52.943569 5561 raft_consensus.cc:738] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Initialized, Role: FOLLOWER
I20250815 07:58:52.944041 5561 consensus_queue.cc:260] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:52.945587 5561 ts_tablet_manager.cc:1428] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250815 07:58:53.109788 5672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:53.110242 5672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:53.110689 5672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:53.139272 5672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:53.140084 5672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:58:53.172497 5672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:37329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=42825
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:53.173812 5672 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:53.175545 5672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:53.187273 5685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:54.309163 5691 raft_consensus.cc:491] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:58:54.309911 5691 raft_consensus.cc:513] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:54.321203 5691 leader_election.cc:290] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941), eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329)
W20250815 07:58:54.322861 5556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.194:37329: connect: Connection refused (error 111)
W20250815 07:58:54.329648 5556 leader_election.cc:336] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329): Network error: Client connection negotiation failed: client connection to 127.1.121.194:37329: connect: Connection refused (error 111)
W20250815 07:58:54.337045 5558 leader_election.cc:336] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Network error: Client connection negotiation failed: client connection to 127.1.121.195:43941: connect: Connection refused (error 111)
I20250815 07:58:54.337615 5558 leader_election.cc:304] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cb7f777e88841a3a4aa5b37a1ab2b95; no voters: bd1eea6c51dc4358a6e5cddd3cb203ee, eb9b068c8b91482fb5236dccf294f0d4
I20250815 07:58:54.338539 5691 raft_consensus.cc:2747] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250815 07:58:54.431207 5691 raft_consensus.cc:491] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:58:54.431779 5691 raft_consensus.cc:513] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:54.437072 5691 leader_election.cc:290] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329), bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941)
W20250815 07:58:54.440124 5556 leader_election.cc:336] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329): Network error: Client connection negotiation failed: client connection to 127.1.121.194:37329: connect: Connection refused (error 111)
W20250815 07:58:54.445861 5558 leader_election.cc:336] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Network error: Client connection negotiation failed: client connection to 127.1.121.195:43941: connect: Connection refused (error 111)
I20250815 07:58:54.446530 5558 leader_election.cc:304] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cb7f777e88841a3a4aa5b37a1ab2b95; no voters: bd1eea6c51dc4358a6e5cddd3cb203ee, eb9b068c8b91482fb5236dccf294f0d4
I20250815 07:58:54.447333 5691 raft_consensus.cc:2747] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250815 07:58:53.188635 5686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:54.713760 5672 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.525s user 0.484s sys 1.035s
W20250815 07:58:54.714145 5672 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.526s user 0.484s sys 1.035s
W20250815 07:58:54.590030 5684 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 5672
W20250815 07:58:54.721537 5687 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1531 milliseconds
W20250815 07:58:54.722222 5688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:54.722257 5672 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:54.723378 5672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:58:54.725378 5672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:54.726696 5672 hybrid_clock.cc:648] HybridClock initialized: now 1755244734726660 us; error 36 us; skew 500 ppm
I20250815 07:58:54.727376 5672 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:54.732828 5672 webserver.cc:480] Webserver started at http://127.1.121.194:42825/ using document root <none> and password file <none>
I20250815 07:58:54.733686 5672 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:54.733899 5672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:54.740968 5672 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20250815 07:58:54.747236 5699 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:54.748353 5672 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250815 07:58:54.748697 5672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "eb9b068c8b91482fb5236dccf294f0d4"
format_stamp: "Formatted at 2025-08-15 07:58:27 on dist-test-slave-dlgj"
I20250815 07:58:54.751298 5672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:54.813650 5672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:54.814921 5672 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:54.815282 5672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:54.817387 5672 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:54.822247 5706 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250815 07:58:54.835589 5672 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250815 07:58:54.835789 5672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.001s
I20250815 07:58:54.836088 5672 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250815 07:58:54.840477 5706 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap starting.
I20250815 07:58:54.845325 5672 ts_tablet_manager.cc:610] Registered 3 tablets
I20250815 07:58:54.845495 5672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.008s sys 0.000s
I20250815 07:58:54.886695 5706 log.cc:826] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:54.987892 5672 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:37329
I20250815 07:58:54.988070 5813 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:37329 every 8 connection(s)
I20250815 07:58:54.990896 5672 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 07:58:54.991762 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 5672
I20250815 07:58:54.996479 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:43941
--local_ip_for_outbound_sockets=127.1.121.195
--tserver_master_addrs=127.1.121.254:46565
--webserver_port=40185
--webserver_interface=127.1.121.195
--builtin_ntp_servers=127.1.121.212:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250815 07:58:55.019798 5706 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:55.020961 5706 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap complete.
I20250815 07:58:55.023414 5706 ts_tablet_manager.cc:1397] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Time spent bootstrapping tablet: real 0.183s user 0.157s sys 0.021s
I20250815 07:58:55.039105 5814 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:55.039521 5814 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:55.040585 5814 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:55.039366 5706 raft_consensus.cc:357] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:55.041918 5706 raft_consensus.cc:738] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Initialized, Role: FOLLOWER
I20250815 07:58:55.042482 5706 consensus_queue.cc:260] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:55.045852 5706 ts_tablet_manager.cc:1428] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4: Time spent starting tablet: real 0.022s user 0.018s sys 0.004s
I20250815 07:58:55.046550 5706 tablet_bootstrap.cc:492] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap starting.
I20250815 07:58:55.050626 5482 ts_manager.cc:194] Registered new tserver with Master: eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329)
I20250815 07:58:55.055342 5482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:46311
I20250815 07:58:55.058683 5814 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
I20250815 07:58:55.127440 5706 tablet_bootstrap.cc:492] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:55.128139 5706 tablet_bootstrap.cc:492] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap complete.
I20250815 07:58:55.129192 5706 ts_tablet_manager.cc:1397] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Time spent bootstrapping tablet: real 0.083s user 0.069s sys 0.012s
I20250815 07:58:55.130560 5706 raft_consensus.cc:357] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:55.130862 5706 raft_consensus.cc:738] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Initialized, Role: FOLLOWER
I20250815 07:58:55.131266 5706 consensus_queue.cc:260] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:55.131605 5706 raft_consensus.cc:397] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:58:55.131808 5706 raft_consensus.cc:491] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:58:55.132097 5706 raft_consensus.cc:3058] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Advancing to term 3
I20250815 07:58:55.136617 5706 raft_consensus.cc:513] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:55.137125 5706 leader_election.cc:304] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9b068c8b91482fb5236dccf294f0d4; no voters:
I20250815 07:58:55.137571 5706 leader_election.cc:290] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250815 07:58:55.137776 5819 raft_consensus.cc:2802] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 07:58:55.140089 5819 raft_consensus.cc:695] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [term 3 LEADER]: Becoming Leader. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Running, Role: LEADER
I20250815 07:58:55.141095 5819 consensus_queue.cc:237] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } }
I20250815 07:58:55.142581 5706 ts_tablet_manager.cc:1428] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20250815 07:58:55.143342 5706 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap starting.
I20250815 07:58:55.150103 5482 catalog_manager.cc:5582] T cb3af857207e4386ad2828460e9a90db P eb9b068c8b91482fb5236dccf294f0d4 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "eb9b068c8b91482fb5236dccf294f0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } health_report { overall_health: HEALTHY } } }
I20250815 07:58:55.267318 5706 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:55.267890 5706 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Bootstrap complete.
I20250815 07:58:55.268909 5706 ts_tablet_manager.cc:1397] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Time spent bootstrapping tablet: real 0.126s user 0.108s sys 0.015s
I20250815 07:58:55.270157 5706 raft_consensus.cc:357] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:55.270548 5706 raft_consensus.cc:738] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9b068c8b91482fb5236dccf294f0d4, State: Initialized, Role: FOLLOWER
I20250815 07:58:55.270975 5706 consensus_queue.cc:260] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:55.272138 5706 ts_tablet_manager.cc:1428] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20250815 07:58:55.314898 5818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:58:55.315322 5818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:58:55.315824 5818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:58:55.342612 5818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:58:55.343425 5818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:58:55.372555 5818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:43941
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=40185
--tserver_master_addrs=127.1.121.254:46565
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:58:55.373811 5818 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:58:55.375209 5818 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:58:55.385396 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:58:56.087641 5841 raft_consensus.cc:491] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:58:56.089063 5841 raft_consensus.cc:513] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:56.091696 5841 leader_election.cc:290] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941), eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329)
W20250815 07:58:56.142783 5558 leader_election.cc:336] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Network error: Client connection negotiation failed: client connection to 127.1.121.195:43941: connect: Connection refused (error 111)
I20250815 07:58:56.149243 5769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47b2ca7263944000837d35652b69ec59" candidate_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "eb9b068c8b91482fb5236dccf294f0d4" is_pre_election: true
I20250815 07:58:56.150242 5769 raft_consensus.cc:2466] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb7f777e88841a3a4aa5b37a1ab2b95 in term 2.
I20250815 07:58:56.152094 5841 raft_consensus.cc:491] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:58:56.152326 5556 leader_election.cc:304] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cb7f777e88841a3a4aa5b37a1ab2b95, eb9b068c8b91482fb5236dccf294f0d4; no voters: bd1eea6c51dc4358a6e5cddd3cb203ee
I20250815 07:58:56.152694 5841 raft_consensus.cc:513] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:56.155182 5841 leader_election.cc:290] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329), bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941)
I20250815 07:58:56.155779 5841 raft_consensus.cc:2802] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250815 07:58:56.156136 5841 raft_consensus.cc:491] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 07:58:56.156569 5841 raft_consensus.cc:3058] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Advancing to term 3
W20250815 07:58:56.164232 5558 leader_election.cc:336] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Network error: Client connection negotiation failed: client connection to 127.1.121.195:43941: connect: Connection refused (error 111)
I20250815 07:58:56.164738 5769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3150c850797343f4bbab2cc804e4bc88" candidate_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "eb9b068c8b91482fb5236dccf294f0d4" is_pre_election: true
I20250815 07:58:56.165416 5769 raft_consensus.cc:2466] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb7f777e88841a3a4aa5b37a1ab2b95 in term 2.
I20250815 07:58:56.166869 5556 leader_election.cc:304] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cb7f777e88841a3a4aa5b37a1ab2b95, eb9b068c8b91482fb5236dccf294f0d4; no voters: bd1eea6c51dc4358a6e5cddd3cb203ee
I20250815 07:58:56.168319 5846 raft_consensus.cc:2802] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250815 07:58:56.168810 5846 raft_consensus.cc:491] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 07:58:56.169279 5846 raft_consensus.cc:3058] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 2 FOLLOWER]: Advancing to term 3
I20250815 07:58:56.177883 5846 raft_consensus.cc:513] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:56.181056 5769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3150c850797343f4bbab2cc804e4bc88" candidate_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "eb9b068c8b91482fb5236dccf294f0d4"
I20250815 07:58:56.181706 5769 raft_consensus.cc:3058] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Advancing to term 3
I20250815 07:58:56.190177 5769 raft_consensus.cc:2466] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb7f777e88841a3a4aa5b37a1ab2b95 in term 3.
I20250815 07:58:56.191089 5846 leader_election.cc:290] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 election: Requested vote from peers eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329), bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941)
I20250815 07:58:56.191989 5556 leader_election.cc:304] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb7f777e88841a3a4aa5b37a1ab2b95, eb9b068c8b91482fb5236dccf294f0d4; no voters:
W20250815 07:58:56.195644 5558 leader_election.cc:336] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Network error: Client connection negotiation failed: client connection to 127.1.121.195:43941: connect: Connection refused (error 111)
I20250815 07:58:56.198136 5846 raft_consensus.cc:2802] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 07:58:56.199335 5846 raft_consensus.cc:695] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 3 LEADER]: Becoming Leader. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Running, Role: LEADER
I20250815 07:58:56.201094 5846 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:56.171586 5841 raft_consensus.cc:513] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:56.209298 5841 leader_election.cc:290] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 election: Requested vote from peers bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941), eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329)
I20250815 07:58:56.211405 5769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47b2ca7263944000837d35652b69ec59" candidate_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "eb9b068c8b91482fb5236dccf294f0d4"
I20250815 07:58:56.211972 5769 raft_consensus.cc:3058] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 2 FOLLOWER]: Advancing to term 3
I20250815 07:58:56.219070 5769 raft_consensus.cc:2466] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb7f777e88841a3a4aa5b37a1ab2b95 in term 3.
W20250815 07:58:56.212810 5558 leader_election.cc:336] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Network error: Client connection negotiation failed: client connection to 127.1.121.195:43941: connect: Connection refused (error 111)
I20250815 07:58:56.222965 5556 leader_election.cc:304] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cb7f777e88841a3a4aa5b37a1ab2b95, eb9b068c8b91482fb5236dccf294f0d4; no voters: bd1eea6c51dc4358a6e5cddd3cb203ee
I20250815 07:58:56.224350 5846 raft_consensus.cc:2802] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 3 FOLLOWER]: Leader election won for term 3
I20250815 07:58:56.225132 5846 raft_consensus.cc:695] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 3 LEADER]: Becoming Leader. State: Replica: 2cb7f777e88841a3a4aa5b37a1ab2b95, State: Running, Role: LEADER
I20250815 07:58:56.226056 5846 consensus_queue.cc:237] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:56.230255 5482 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: term changed from 2 to 3, leader changed from <none> to 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193). New cstate: current_term: 3 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250815 07:58:56.260413 5482 catalog_manager.cc:5582] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: term changed from 2 to 3, leader changed from bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) to 2cb7f777e88841a3a4aa5b37a1ab2b95 (127.1.121.193). New cstate: current_term: 3 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250815 07:58:56.666379 5558 consensus_peers.cc:489] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 -> Peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Couldn't send request to peer bd1eea6c51dc4358a6e5cddd3cb203ee. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.195:43941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 07:58:56.686070 5769 raft_consensus.cc:1273] T 47b2ca7263944000837d35652b69ec59 P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Refusing update from remote peer 2cb7f777e88841a3a4aa5b37a1ab2b95: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250815 07:58:56.688441 5846 consensus_queue.cc:1035] T 47b2ca7263944000837d35652b69ec59 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20250815 07:58:55.386974 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:56.737711 5837 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1350 milliseconds
W20250815 07:58:56.739362 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:58:56.741631 5818 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.355s user 0.004s sys 0.000s
W20250815 07:58:56.741856 5818 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.355s user 0.004s sys 0.000s
I20250815 07:58:56.742055 5818 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:58:56.743136 5818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20250815 07:58:56.747850 5558 consensus_peers.cc:489] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 -> Peer bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): Couldn't send request to peer bd1eea6c51dc4358a6e5cddd3cb203ee. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.195:43941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 07:58:56.756659 5818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:58:56.758162 5818 hybrid_clock.cc:648] HybridClock initialized: now 1755244736758093 us; error 62 us; skew 500 ppm
I20250815 07:58:56.758883 5818 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:58:56.767925 5818 webserver.cc:480] Webserver started at http://127.1.121.195:40185/ using document root <none> and password file <none>
I20250815 07:58:56.768882 5818 fs_manager.cc:362] Metadata directory not provided
I20250815 07:58:56.769083 5818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:58:56.785501 5818 fs_manager.cc:714] Time spent opening directory manager: real 0.013s user 0.012s sys 0.000s
I20250815 07:58:56.798581 5862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:58:56.799829 5818 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.001s sys 0.004s
I20250815 07:58:56.800211 5818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee"
format_stamp: "Formatted at 2025-08-15 07:58:29 on dist-test-slave-dlgj"
I20250815 07:58:56.802599 5769 raft_consensus.cc:1273] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Refusing update from remote peer 2cb7f777e88841a3a4aa5b37a1ab2b95: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250815 07:58:56.802829 5818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:58:56.804096 5841 consensus_queue.cc:1035] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250815 07:58:56.845443 5624 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 14, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:56.848676 5769 raft_consensus.cc:1273] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Refusing update from remote peer 2cb7f777e88841a3a4aa5b37a1ab2b95: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20250815 07:58:56.849540 5846 consensus_queue.cc:1035] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250815 07:58:56.853435 5841 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 14 to 16, VOTER bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:56.857090 5769 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Committing config change with OpId 3.16: config changed from index 14 to 16, VOTER bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } }
I20250815 07:58:56.861086 5469 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3150c850797343f4bbab2cc804e4bc88 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 07:58:56.865584 5481 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: config changed from index 14 to 16, VOTER bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195) evicted. New cstate: current_term: 3 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250815 07:58:56.883745 5481 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 3150c850797343f4bbab2cc804e4bc88 on TS bd1eea6c51dc4358a6e5cddd3cb203ee: Not found: failed to reset TS proxy: Could not find TS for UUID bd1eea6c51dc4358a6e5cddd3cb203ee
I20250815 07:58:56.885754 5818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:58:56.886132 5624 consensus_queue.cc:237] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } }
I20250815 07:58:56.887573 5818 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:58:56.888067 5818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:58:56.888005 5841 raft_consensus.cc:2953] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 [term 3 LEADER]: Committing config change with OpId 3.17: config changed from index 16 to 17, VOTER eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) evicted. New config: { opid_index: 17 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } }
I20250815 07:58:56.892047 5818 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:58:56.893930 5469 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3150c850797343f4bbab2cc804e4bc88 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 07:58:56.897270 5482 catalog_manager.cc:5582] T 3150c850797343f4bbab2cc804e4bc88 P 2cb7f777e88841a3a4aa5b37a1ab2b95 reported cstate change: config changed from index 16 to 17, VOTER eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194) evicted. New cstate: current_term: 3 leader_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" committed_config { opid_index: 17 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } health_report { overall_health: HEALTHY } } }
I20250815 07:58:56.897908 5875 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250815 07:58:56.910060 5818 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250815 07:58:56.910236 5818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.000s
I20250815 07:58:56.910470 5818 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250815 07:58:56.913050 5749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3150c850797343f4bbab2cc804e4bc88 with delete_type TABLET_DATA_TOMBSTONED (TS eb9b068c8b91482fb5236dccf294f0d4 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:48120
I20250815 07:58:56.915346 5875 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap starting.
I20250815 07:58:56.915462 5877 tablet_replica.cc:331] stopping tablet replica
I20250815 07:58:56.916028 5877 raft_consensus.cc:2241] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250815 07:58:56.916472 5877 raft_consensus.cc:2270] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250815 07:58:56.918725 5877 ts_tablet_manager.cc:1905] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 07:58:56.919582 5818 ts_tablet_manager.cc:610] Registered 2 tablets
I20250815 07:58:56.919824 5818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.006s sys 0.000s
I20250815 07:58:56.928882 5877 ts_tablet_manager.cc:1918] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.16
I20250815 07:58:56.929193 5877 log.cc:1199] T 3150c850797343f4bbab2cc804e4bc88 P eb9b068c8b91482fb5236dccf294f0d4: Deleting WAL directory at /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/wals/3150c850797343f4bbab2cc804e4bc88
I20250815 07:58:56.930436 5467 catalog_manager.cc:4928] TS eb9b068c8b91482fb5236dccf294f0d4 (127.1.121.194:37329): tablet 3150c850797343f4bbab2cc804e4bc88 (table TestTable [id=79af04ffebc249b6a8f733e9cdf514ef]) successfully deleted
W20250815 07:58:56.936221 5467 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 3150c850797343f4bbab2cc804e4bc88 on TS bd1eea6c51dc4358a6e5cddd3cb203ee failed: Not found: failed to reset TS proxy: Could not find TS for UUID bd1eea6c51dc4358a6e5cddd3cb203ee
I20250815 07:58:56.982937 5875 log.cc:826] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Log is configured to *not* fsync() on all Append() calls
I20250815 07:58:57.073786 5818 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:43941
I20250815 07:58:57.074028 5984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:43941 every 8 connection(s)
I20250815 07:58:57.076323 5818 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 07:58:57.077426 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 5818
I20250815 07:58:57.105499 5875 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:57.106441 5875 tablet_bootstrap.cc:492] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap complete.
I20250815 07:58:57.107909 5875 ts_tablet_manager.cc:1397] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent bootstrapping tablet: real 0.193s user 0.156s sys 0.031s
I20250815 07:58:57.115850 5985 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46565
I20250815 07:58:57.116218 5985 heartbeater.cc:461] Registering TS with master...
I20250815 07:58:57.117027 5985 heartbeater.cc:507] Master 127.1.121.254:46565 requested a full tablet report, sending...
I20250815 07:58:57.119846 5482 ts_manager.cc:194] Registered new tserver with Master: bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941)
I20250815 07:58:57.119867 5875 raft_consensus.cc:357] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:57.121798 5875 raft_consensus.cc:738] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1eea6c51dc4358a6e5cddd3cb203ee, State: Initialized, Role: FOLLOWER
I20250815 07:58:57.122375 5875 consensus_queue.cc:260] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } } peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } attrs { promote: false } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } }
I20250815 07:58:57.122792 5482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:50169
I20250815 07:58:57.125694 5875 ts_tablet_manager.cc:1428] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent starting tablet: real 0.018s user 0.015s sys 0.001s
I20250815 07:58:57.126325 5985 heartbeater.cc:499] Master 127.1.121.254:46565 was elected leader, sending a full tablet report...
I20250815 07:58:57.126345 5875 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap starting.
I20250815 07:58:57.126684 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:58:57.130532 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250815 07:58:57.133246 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250815 07:58:57.226101 5940 raft_consensus.cc:3058] T 47b2ca7263944000837d35652b69ec59 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Advancing to term 3
I20250815 07:58:57.228674 5875 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 07:58:57.229269 5875 tablet_bootstrap.cc:492] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Bootstrap complete.
I20250815 07:58:57.230338 5875 ts_tablet_manager.cc:1397] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent bootstrapping tablet: real 0.104s user 0.085s sys 0.016s
I20250815 07:58:57.231613 5875 raft_consensus.cc:357] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:57.231985 5875 raft_consensus.cc:738] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1eea6c51dc4358a6e5cddd3cb203ee, State: Initialized, Role: FOLLOWER
I20250815 07:58:57.232501 5875 consensus_queue.cc:260] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "2cb7f777e88841a3a4aa5b37a1ab2b95" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 37599 } } peers { permanent_uuid: "eb9b068c8b91482fb5236dccf294f0d4" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 37329 } attrs { promote: false } } peers { permanent_uuid: "bd1eea6c51dc4358a6e5cddd3cb203ee" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 43941 } attrs { promote: false } }
I20250815 07:58:57.233685 5875 ts_tablet_manager.cc:1428] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20250815 07:58:57.264716 5919 tablet_service.cc:1515] Processing DeleteTablet for tablet 3150c850797343f4bbab2cc804e4bc88 with delete_type TABLET_DATA_TOMBSTONED (TS bd1eea6c51dc4358a6e5cddd3cb203ee not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:43234
I20250815 07:58:57.268714 5996 tablet_replica.cc:331] stopping tablet replica
I20250815 07:58:57.269192 5996 raft_consensus.cc:2241] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 07:58:57.269539 5996 raft_consensus.cc:2270] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 07:58:57.271857 5996 ts_tablet_manager.cc:1905] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 07:58:57.280221 5996 ts_tablet_manager.cc:1918] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.14
I20250815 07:58:57.280468 5996 log.cc:1199] T 3150c850797343f4bbab2cc804e4bc88 P bd1eea6c51dc4358a6e5cddd3cb203ee: Deleting WAL directory at /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/wals/3150c850797343f4bbab2cc804e4bc88
I20250815 07:58:57.281615 5469 catalog_manager.cc:4928] TS bd1eea6c51dc4358a6e5cddd3cb203ee (127.1.121.195:43941): tablet 3150c850797343f4bbab2cc804e4bc88 (table TestTable [id=79af04ffebc249b6a8f733e9cdf514ef]) successfully deleted
W20250815 07:58:58.136569 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:58:58.314304 5513 debug-util.cc:398] Leaking SignalData structure 0x7b08000a95e0 after lost signal to thread 5450
W20250815 07:58:58.314884 5513 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9060 after lost signal to thread 5516
W20250815 07:58:59.139752 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:00.143574 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:01.147099 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:02.150343 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:03.153656 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:04.156775 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:05.160039 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:06.163260 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:07.166515 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:08.169971 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:09.173143 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:10.177031 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:11.180747 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:12.183744 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:13.186851 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:14.189849 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:15.192838 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250815 07:59:16.195943 1511 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 3150c850797343f4bbab2cc804e4bc88: tablet_id: "3150c850797343f4bbab2cc804e4bc88" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250815 07:59:17.198812 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 5519
I20250815 07:59:17.222676 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 5672
I20250815 07:59:17.246989 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 5818
I20250815 07:59:17.270018 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 5448
2025-08-15T07:59:17Z chronyd exiting
I20250815 07:59:17.313978 1511 test_util.cc:183] -----------------------------------------------
I20250815 07:59:17.314157 1511 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755244639110353-1511-0
[ FAILED ] AdminCliTest.TestRebuildTables (56372 ms)
[----------] 5 tests from AdminCliTest (118141 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250815 07:59:17.317623 1511 test_util.cc:276] Using random seed: -1002375199
I20250815 07:59:17.321714 1511 ts_itest-base.cc:115] Starting cluster with:
I20250815 07:59:17.321841 1511 ts_itest-base.cc:116] --------------
I20250815 07:59:17.321997 1511 ts_itest-base.cc:117] 5 tablet servers
I20250815 07:59:17.322124 1511 ts_itest-base.cc:118] 3 replicas per TS
I20250815 07:59:17.322252 1511 ts_itest-base.cc:119] --------------
2025-08-15T07:59:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T07:59:17Z Disabled control of system clock
I20250815 07:59:17.357714 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46039
--webserver_interface=127.1.121.254
--webserver_port=0
--builtin_ntp_servers=127.1.121.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:46039
--raft_prepare_replacement_before_eviction=true with env {}
W20250815 07:59:17.618201 6017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:17.618777 6017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:17.619284 6017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:17.645227 6017 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 07:59:17.645581 6017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:59:17.645815 6017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:17.646063 6017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:59:17.646293 6017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:59:17.674938 6017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:46039
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:46039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:17.676064 6017 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:17.677520 6017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:17.686348 6023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:19.090304 6022 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 6017
W20250815 07:59:19.269078 6017 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.582s user 0.603s sys 0.979s
W20250815 07:59:17.687734 6024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:19.270272 6017 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.583s user 0.603s sys 0.979s
W20250815 07:59:19.270764 6025 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1582 milliseconds
I20250815 07:59:19.271471 6017 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250815 07:59:19.271528 6026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:19.274327 6017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:19.276530 6017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:19.277861 6017 hybrid_clock.cc:648] HybridClock initialized: now 1755244759277827 us; error 41 us; skew 500 ppm
I20250815 07:59:19.278534 6017 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:19.283703 6017 webserver.cc:480] Webserver started at http://127.1.121.254:35047/ using document root <none> and password file <none>
I20250815 07:59:19.284511 6017 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:19.284694 6017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:19.285072 6017 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:19.288946 6017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "93e934e09537409c92564ba0ea7bdaa1"
format_stamp: "Formatted at 2025-08-15 07:59:19 on dist-test-slave-dlgj"
I20250815 07:59:19.289858 6017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "93e934e09537409c92564ba0ea7bdaa1"
format_stamp: "Formatted at 2025-08-15 07:59:19 on dist-test-slave-dlgj"
I20250815 07:59:19.295802 6017 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20250815 07:59:19.300599 6033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:19.301515 6017 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250815 07:59:19.301777 6017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "93e934e09537409c92564ba0ea7bdaa1"
format_stamp: "Formatted at 2025-08-15 07:59:19 on dist-test-slave-dlgj"
I20250815 07:59:19.302038 6017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:19.347563 6017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:19.348796 6017 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:19.349146 6017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:19.411293 6017 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:46039
I20250815 07:59:19.411372 6084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:46039 every 8 connection(s)
I20250815 07:59:19.413779 6017 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:59:19.418453 6085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:19.423313 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 6017
I20250815 07:59:19.423843 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 07:59:19.443259 6085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1: Bootstrap starting.
I20250815 07:59:19.448980 6085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:19.450502 6085 log.cc:826] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:19.454110 6085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1: No bootstrap required, opened a new log
I20250815 07:59:19.468582 6085 raft_consensus.cc:357] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e934e09537409c92564ba0ea7bdaa1" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46039 } }
I20250815 07:59:19.469074 6085 raft_consensus.cc:383] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:19.469275 6085 raft_consensus.cc:738] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e934e09537409c92564ba0ea7bdaa1, State: Initialized, Role: FOLLOWER
I20250815 07:59:19.469820 6085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e934e09537409c92564ba0ea7bdaa1" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46039 } }
I20250815 07:59:19.470225 6085 raft_consensus.cc:397] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:59:19.470510 6085 raft_consensus.cc:491] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:59:19.470870 6085 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:19.474319 6085 raft_consensus.cc:513] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e934e09537409c92564ba0ea7bdaa1" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46039 } }
I20250815 07:59:19.474888 6085 leader_election.cc:304] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93e934e09537409c92564ba0ea7bdaa1; no voters:
I20250815 07:59:19.476411 6085 leader_election.cc:290] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:59:19.476999 6091 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:59:19.479266 6091 raft_consensus.cc:695] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [term 1 LEADER]: Becoming Leader. State: Replica: 93e934e09537409c92564ba0ea7bdaa1, State: Running, Role: LEADER
I20250815 07:59:19.479933 6091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e934e09537409c92564ba0ea7bdaa1" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46039 } }
I20250815 07:59:19.480459 6085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:59:19.489370 6092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93e934e09537409c92564ba0ea7bdaa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e934e09537409c92564ba0ea7bdaa1" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46039 } } }
I20250815 07:59:19.490166 6092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [sys.catalog]: This master's current role is: LEADER
I20250815 07:59:19.491606 6093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e934e09537409c92564ba0ea7bdaa1. Latest consensus state: current_term: 1 leader_uuid: "93e934e09537409c92564ba0ea7bdaa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e934e09537409c92564ba0ea7bdaa1" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 46039 } } }
I20250815 07:59:19.492338 6093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1 [sys.catalog]: This master's current role is: LEADER
I20250815 07:59:19.493100 6099 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:59:19.501755 6099 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:59:19.515566 6099 catalog_manager.cc:1349] Generated new cluster ID: 2770b37eec6747aa976b3de872cf0810
I20250815 07:59:19.515848 6099 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:59:19.545606 6099 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 07:59:19.546844 6099 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:59:19.558063 6099 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 93e934e09537409c92564ba0ea7bdaa1: Generated new TSK 0
I20250815 07:59:19.558871 6099 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:59:19.573099 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:0
--local_ip_for_outbound_sockets=127.1.121.193
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--builtin_ntp_servers=127.1.121.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250815 07:59:19.841148 6110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:19.841624 6110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:19.842079 6110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:19.868587 6110 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 07:59:19.868897 6110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:19.869555 6110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:59:19.901278 6110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:19.902446 6110 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:19.903885 6110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:19.915380 6116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:19.917922 6117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:21.215682 6118 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1297 milliseconds
W20250815 07:59:21.216320 6119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:21.216909 6110 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.299s user 0.446s sys 0.745s
W20250815 07:59:21.217228 6110 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.299s user 0.446s sys 0.745s
I20250815 07:59:21.217487 6110 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:21.218823 6110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:21.228471 6110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:21.229928 6110 hybrid_clock.cc:648] HybridClock initialized: now 1755244761229855 us; error 65 us; skew 500 ppm
I20250815 07:59:21.230954 6110 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:21.238965 6110 webserver.cc:480] Webserver started at http://127.1.121.193:39293/ using document root <none> and password file <none>
I20250815 07:59:21.240226 6110 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:21.240504 6110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:21.241088 6110 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:21.247484 6110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cd8769261c8345f7bff7ef619e476aae"
format_stamp: "Formatted at 2025-08-15 07:59:21 on dist-test-slave-dlgj"
I20250815 07:59:21.248828 6110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cd8769261c8345f7bff7ef619e476aae"
format_stamp: "Formatted at 2025-08-15 07:59:21 on dist-test-slave-dlgj"
I20250815 07:59:21.257678 6110 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250815 07:59:21.264551 6126 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:21.265631 6110 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 07:59:21.265997 6110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cd8769261c8345f7bff7ef619e476aae"
format_stamp: "Formatted at 2025-08-15 07:59:21 on dist-test-slave-dlgj"
I20250815 07:59:21.266409 6110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:21.342619 6110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:21.344457 6110 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:21.344928 6110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:21.348155 6110 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:21.352320 6110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:21.352509 6110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:21.352736 6110 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:21.352883 6110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:21.496794 6110 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:35299
I20250815 07:59:21.496861 6238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:35299 every 8 connection(s)
I20250815 07:59:21.500084 6110 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:59:21.505722 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 6110
I20250815 07:59:21.506193 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 07:59:21.513548 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:0
--local_ip_for_outbound_sockets=127.1.121.194
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--builtin_ntp_servers=127.1.121.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 07:59:21.531599 6239 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46039
I20250815 07:59:21.532066 6239 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:21.533190 6239 heartbeater.cc:507] Master 127.1.121.254:46039 requested a full tablet report, sending...
I20250815 07:59:21.536116 6050 ts_manager.cc:194] Registered new tserver with Master: cd8769261c8345f7bff7ef619e476aae (127.1.121.193:35299)
I20250815 07:59:21.538908 6050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:58181
W20250815 07:59:21.785207 6243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:21.785580 6243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:21.785984 6243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:21.811564 6243 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 07:59:21.811834 6243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:21.812515 6243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:59:21.841114 6243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:21.842181 6243 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:21.843477 6243 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:21.853796 6249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:22.542816 6239 heartbeater.cc:499] Master 127.1.121.254:46039 was elected leader, sending a full tablet report...
W20250815 07:59:21.854269 6250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:22.989828 6252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:22.992548 6251 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1133 milliseconds
W20250815 07:59:22.995776 6243 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.142s user 0.324s sys 0.806s
W20250815 07:59:22.996047 6243 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.142s user 0.325s sys 0.806s
I20250815 07:59:22.996263 6243 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:22.997268 6243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:22.999153 6243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:23.000460 6243 hybrid_clock.cc:648] HybridClock initialized: now 1755244763000419 us; error 46 us; skew 500 ppm
I20250815 07:59:23.001179 6243 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:23.007349 6243 webserver.cc:480] Webserver started at http://127.1.121.194:46453/ using document root <none> and password file <none>
I20250815 07:59:23.008203 6243 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:23.008369 6243 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:23.008770 6243 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:23.012475 6243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "27498835651749758f90684f6cfe6f01"
format_stamp: "Formatted at 2025-08-15 07:59:23 on dist-test-slave-dlgj"
I20250815 07:59:23.013387 6243 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "27498835651749758f90684f6cfe6f01"
format_stamp: "Formatted at 2025-08-15 07:59:23 on dist-test-slave-dlgj"
I20250815 07:59:23.020145 6243 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250815 07:59:23.025257 6259 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:23.026242 6243 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250815 07:59:23.026504 6243 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "27498835651749758f90684f6cfe6f01"
format_stamp: "Formatted at 2025-08-15 07:59:23 on dist-test-slave-dlgj"
I20250815 07:59:23.026790 6243 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:23.089532 6243 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:23.090790 6243 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:23.091172 6243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:23.093459 6243 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:23.097214 6243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:23.097422 6243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:23.097651 6243 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:23.097805 6243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:23.224143 6243 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:41317
I20250815 07:59:23.224263 6371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:41317 every 8 connection(s)
I20250815 07:59:23.227290 6243 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250815 07:59:23.230563 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 6243
I20250815 07:59:23.231060 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250815 07:59:23.238384 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:0
--local_ip_for_outbound_sockets=127.1.121.195
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--builtin_ntp_servers=127.1.121.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 07:59:23.250036 6372 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46039
I20250815 07:59:23.250384 6372 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:23.251231 6372 heartbeater.cc:507] Master 127.1.121.254:46039 requested a full tablet report, sending...
I20250815 07:59:23.253142 6050 ts_manager.cc:194] Registered new tserver with Master: 27498835651749758f90684f6cfe6f01 (127.1.121.194:41317)
I20250815 07:59:23.254813 6050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:41937
W20250815 07:59:23.513208 6376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:23.513561 6376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:23.513922 6376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:23.540588 6376 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 07:59:23.540915 6376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:23.541561 6376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:59:23.571842 6376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:23.572875 6376 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:23.574187 6376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:23.584762 6382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:24.257688 6372 heartbeater.cc:499] Master 127.1.121.254:46039 was elected leader, sending a full tablet report...
W20250815 07:59:23.585373 6383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:23.587976 6385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:24.628721 6384 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250815 07:59:24.628760 6376 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:24.631919 6376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:24.633842 6376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:24.635128 6376 hybrid_clock.cc:648] HybridClock initialized: now 1755244764635096 us; error 48 us; skew 500 ppm
I20250815 07:59:24.635834 6376 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:24.641642 6376 webserver.cc:480] Webserver started at http://127.1.121.195:45471/ using document root <none> and password file <none>
I20250815 07:59:24.642436 6376 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:24.642614 6376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:24.643008 6376 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:24.646710 6376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7d31ddf2b8944dc7969ca0ce95ceaa7e"
format_stamp: "Formatted at 2025-08-15 07:59:24 on dist-test-slave-dlgj"
I20250815 07:59:24.647639 6376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7d31ddf2b8944dc7969ca0ce95ceaa7e"
format_stamp: "Formatted at 2025-08-15 07:59:24 on dist-test-slave-dlgj"
I20250815 07:59:24.653575 6376 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20250815 07:59:24.658214 6392 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:24.659065 6376 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20250815 07:59:24.659317 6376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7d31ddf2b8944dc7969ca0ce95ceaa7e"
format_stamp: "Formatted at 2025-08-15 07:59:24 on dist-test-slave-dlgj"
I20250815 07:59:24.659605 6376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:24.718936 6376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:24.720335 6376 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:24.720738 6376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:24.722975 6376 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:24.726529 6376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:24.726727 6376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:24.726964 6376 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:24.727113 6376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:24.846611 6376 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:37847
I20250815 07:59:24.846709 6504 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:37847 every 8 connection(s)
I20250815 07:59:24.848973 6376 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250815 07:59:24.852125 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 6376
I20250815 07:59:24.852640 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250815 07:59:24.858613 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.196:0
--local_ip_for_outbound_sockets=127.1.121.196
--webserver_interface=127.1.121.196
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--builtin_ntp_servers=127.1.121.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 07:59:24.867240 6505 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46039
I20250815 07:59:24.867583 6505 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:24.868593 6505 heartbeater.cc:507] Master 127.1.121.254:46039 requested a full tablet report, sending...
I20250815 07:59:24.870450 6050 ts_manager.cc:194] Registered new tserver with Master: 7d31ddf2b8944dc7969ca0ce95ceaa7e (127.1.121.195:37847)
I20250815 07:59:24.871579 6050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:53041
W20250815 07:59:25.124131 6509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:25.124575 6509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:25.125036 6509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:25.153653 6509 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 07:59:25.154017 6509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:25.154757 6509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.196
I20250815 07:59:25.185685 6509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.1.121.196
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.196
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:25.186892 6509 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:25.188360 6509 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:25.198284 6515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:25.874186 6505 heartbeater.cc:499] Master 127.1.121.254:46039 was elected leader, sending a full tablet report...
W20250815 07:59:25.199855 6516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:26.274412 6517 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1073 milliseconds
W20250815 07:59:26.275594 6518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:26.278012 6509 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.079s user 0.001s sys 0.003s
W20250815 07:59:26.278244 6509 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.079s user 0.001s sys 0.003s
I20250815 07:59:26.278422 6509 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:26.279408 6509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:26.281939 6509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:26.283427 6509 hybrid_clock.cc:648] HybridClock initialized: now 1755244766283368 us; error 59 us; skew 500 ppm
I20250815 07:59:26.284214 6509 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:26.291278 6509 webserver.cc:480] Webserver started at http://127.1.121.196:46495/ using document root <none> and password file <none>
I20250815 07:59:26.292158 6509 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:26.292361 6509 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:26.292761 6509 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:26.296707 6509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "724bd023063a4d4c97a6f4dfd68a6dce"
format_stamp: "Formatted at 2025-08-15 07:59:26 on dist-test-slave-dlgj"
I20250815 07:59:26.297700 6509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "724bd023063a4d4c97a6f4dfd68a6dce"
format_stamp: "Formatted at 2025-08-15 07:59:26 on dist-test-slave-dlgj"
I20250815 07:59:26.304787 6509 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250815 07:59:26.310374 6525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:26.311339 6509 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250815 07:59:26.311591 6509 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "724bd023063a4d4c97a6f4dfd68a6dce"
format_stamp: "Formatted at 2025-08-15 07:59:26 on dist-test-slave-dlgj"
I20250815 07:59:26.311856 6509 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:26.376843 6509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:26.378093 6509 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:26.378461 6509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:26.380611 6509 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:26.384150 6509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:26.384337 6509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:26.384542 6509 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:26.384676 6509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:26.508502 6509 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.196:34225
I20250815 07:59:26.508589 6637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.196:34225 every 8 connection(s)
I20250815 07:59:26.510607 6509 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250815 07:59:26.511255 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 6509
I20250815 07:59:26.511590 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250815 07:59:26.518006 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.197:0
--local_ip_for_outbound_sockets=127.1.121.197
--webserver_interface=127.1.121.197
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--builtin_ntp_servers=127.1.121.212:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250815 07:59:26.530102 6638 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46039
I20250815 07:59:26.530433 6638 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:26.531225 6638 heartbeater.cc:507] Master 127.1.121.254:46039 requested a full tablet report, sending...
I20250815 07:59:26.532997 6050 ts_manager.cc:194] Registered new tserver with Master: 724bd023063a4d4c97a6f4dfd68a6dce (127.1.121.196:34225)
I20250815 07:59:26.534034 6050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.196:34313
W20250815 07:59:26.795529 6642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:26.795892 6642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:26.796469 6642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:26.822932 6642 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250815 07:59:26.823230 6642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:26.823884 6642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.197
I20250815 07:59:26.852736 6642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.1.121.197
--webserver_port=0
--tserver_master_addrs=127.1.121.254:46039
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.197
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:26.853785 6642 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:26.855206 6642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:26.865180 6648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:27.536912 6638 heartbeater.cc:499] Master 127.1.121.254:46039 was elected leader, sending a full tablet report...
W20250815 07:59:28.268594 6647 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 6642
W20250815 07:59:26.865648 6649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:28.403364 6647 kernel_stack_watchdog.cc:198] Thread 6642 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 07:59:28.407932 6642 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.542s user 0.576s sys 0.953s
W20250815 07:59:28.408273 6642 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.543s user 0.576s sys 0.953s
W20250815 07:59:28.408948 6650 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1543 milliseconds
W20250815 07:59:28.409590 6651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:28.409626 6642 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:28.410570 6642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:28.416445 6642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:28.417768 6642 hybrid_clock.cc:648] HybridClock initialized: now 1755244768417732 us; error 36 us; skew 500 ppm
I20250815 07:59:28.418502 6642 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:28.423825 6642 webserver.cc:480] Webserver started at http://127.1.121.197:42255/ using document root <none> and password file <none>
I20250815 07:59:28.424679 6642 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:28.424862 6642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:28.425242 6642 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:28.429242 6642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7470d1d8481a419db3adee1f18ab1f7d"
format_stamp: "Formatted at 2025-08-15 07:59:28 on dist-test-slave-dlgj"
I20250815 07:59:28.430145 6642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7470d1d8481a419db3adee1f18ab1f7d"
format_stamp: "Formatted at 2025-08-15 07:59:28 on dist-test-slave-dlgj"
I20250815 07:59:28.435941 6642 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250815 07:59:28.440843 6659 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:28.441707 6642 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250815 07:59:28.441956 6642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7470d1d8481a419db3adee1f18ab1f7d"
format_stamp: "Formatted at 2025-08-15 07:59:28 on dist-test-slave-dlgj"
I20250815 07:59:28.442229 6642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:28.501710 6642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:28.502892 6642 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:28.503260 6642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:28.505388 6642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:28.510298 6642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:28.510528 6642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:28.510793 6642 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:28.510980 6642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:28.633831 6642 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.197:40537
I20250815 07:59:28.633891 6771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.197:40537 every 8 connection(s)
I20250815 07:59:28.636040 6642 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250815 07:59:28.644364 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 6642
I20250815 07:59:28.644704 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250815 07:59:28.658095 6772 heartbeater.cc:344] Connected to a master server at 127.1.121.254:46039
I20250815 07:59:28.658439 6772 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:28.659277 6772 heartbeater.cc:507] Master 127.1.121.254:46039 requested a full tablet report, sending...
I20250815 07:59:28.661079 6050 ts_manager.cc:194] Registered new tserver with Master: 7470d1d8481a419db3adee1f18ab1f7d (127.1.121.197:40537)
I20250815 07:59:28.662472 6050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.197:47247
I20250815 07:59:28.664327 1511 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250815 07:59:28.694672 6049 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41808:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 07:59:28.757445 6307 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0ee758121c40bd8590fc46775492b3 (DEFAULT_TABLE table=TestTable [id=40e982e5f0c344b3bc342029f21a7b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:59:28.758291 6573 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0ee758121c40bd8590fc46775492b3 (DEFAULT_TABLE table=TestTable [id=40e982e5f0c344b3bc342029f21a7b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:59:28.759171 6307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0ee758121c40bd8590fc46775492b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:28.759682 6573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0ee758121c40bd8590fc46775492b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:28.760290 6707 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0ee758121c40bd8590fc46775492b3 (DEFAULT_TABLE table=TestTable [id=40e982e5f0c344b3bc342029f21a7b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:59:28.761896 6707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0ee758121c40bd8590fc46775492b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:28.777668 6791 tablet_bootstrap.cc:492] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce: Bootstrap starting.
I20250815 07:59:28.782711 6791 tablet_bootstrap.cc:654] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:28.784821 6792 tablet_bootstrap.cc:492] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01: Bootstrap starting.
I20250815 07:59:28.786859 6791 log.cc:826] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:28.787047 6793 tablet_bootstrap.cc:492] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d: Bootstrap starting.
I20250815 07:59:28.790755 6792 tablet_bootstrap.cc:654] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:28.791873 6791 tablet_bootstrap.cc:492] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce: No bootstrap required, opened a new log
I20250815 07:59:28.792224 6792 log.cc:826] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:28.792292 6791 ts_tablet_manager.cc:1397] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20250815 07:59:28.793145 6793 tablet_bootstrap.cc:654] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:28.795091 6793 log.cc:826] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:28.796389 6792 tablet_bootstrap.cc:492] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01: No bootstrap required, opened a new log
I20250815 07:59:28.796862 6792 ts_tablet_manager.cc:1397] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250815 07:59:28.800590 6793 tablet_bootstrap.cc:492] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d: No bootstrap required, opened a new log
I20250815 07:59:28.801014 6793 ts_tablet_manager.cc:1397] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20250815 07:59:28.813655 6792 raft_consensus.cc:357] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.814183 6792 raft_consensus.cc:383] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:28.814410 6792 raft_consensus.cc:738] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27498835651749758f90684f6cfe6f01, State: Initialized, Role: FOLLOWER
I20250815 07:59:28.815006 6792 consensus_queue.cc:260] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.815121 6791 raft_consensus.cc:357] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.815845 6791 raft_consensus.cc:383] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:28.816147 6791 raft_consensus.cc:738] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724bd023063a4d4c97a6f4dfd68a6dce, State: Initialized, Role: FOLLOWER
I20250815 07:59:28.816989 6791 consensus_queue.cc:260] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.818534 6792 ts_tablet_manager.cc:1428] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01: Time spent starting tablet: real 0.021s user 0.017s sys 0.004s
I20250815 07:59:28.821518 6791 ts_tablet_manager.cc:1428] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce: Time spent starting tablet: real 0.029s user 0.023s sys 0.004s
I20250815 07:59:28.824963 6793 raft_consensus.cc:357] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.826026 6793 raft_consensus.cc:383] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:28.826283 6793 raft_consensus.cc:738] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7470d1d8481a419db3adee1f18ab1f7d, State: Initialized, Role: FOLLOWER
I20250815 07:59:28.827209 6793 consensus_queue.cc:260] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.829825 6772 heartbeater.cc:499] Master 127.1.121.254:46039 was elected leader, sending a full tablet report...
I20250815 07:59:28.830646 6793 ts_tablet_manager.cc:1428] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d: Time spent starting tablet: real 0.029s user 0.016s sys 0.011s
I20250815 07:59:28.885462 6798 raft_consensus.cc:491] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:59:28.885877 6798 raft_consensus.cc:513] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.888080 6798 leader_election.cc:290] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7470d1d8481a419db3adee1f18ab1f7d (127.1.121.197:40537), 27498835651749758f90684f6cfe6f01 (127.1.121.194:41317)
W20250815 07:59:28.889760 6773 tablet.cc:2378] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:59:28.898705 6727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0ee758121c40bd8590fc46775492b3" candidate_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7470d1d8481a419db3adee1f18ab1f7d" is_pre_election: true
I20250815 07:59:28.899370 6727 raft_consensus.cc:2466] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724bd023063a4d4c97a6f4dfd68a6dce in term 0.
I20250815 07:59:28.899680 6327 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0ee758121c40bd8590fc46775492b3" candidate_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27498835651749758f90684f6cfe6f01" is_pre_election: true
I20250815 07:59:28.900465 6327 raft_consensus.cc:2466] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724bd023063a4d4c97a6f4dfd68a6dce in term 0.
I20250815 07:59:28.900660 6529 leader_election.cc:304] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724bd023063a4d4c97a6f4dfd68a6dce, 7470d1d8481a419db3adee1f18ab1f7d; no voters:
I20250815 07:59:28.901393 6798 raft_consensus.cc:2802] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 07:59:28.901691 6798 raft_consensus.cc:491] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 07:59:28.901969 6798 raft_consensus.cc:3058] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:28.905859 6798 raft_consensus.cc:513] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.907156 6798 leader_election.cc:290] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [CANDIDATE]: Term 1 election: Requested vote from peers 7470d1d8481a419db3adee1f18ab1f7d (127.1.121.197:40537), 27498835651749758f90684f6cfe6f01 (127.1.121.194:41317)
I20250815 07:59:28.907727 6727 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0ee758121c40bd8590fc46775492b3" candidate_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7470d1d8481a419db3adee1f18ab1f7d"
I20250815 07:59:28.907920 6327 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0ee758121c40bd8590fc46775492b3" candidate_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27498835651749758f90684f6cfe6f01"
I20250815 07:59:28.908078 6727 raft_consensus.cc:3058] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:28.908300 6327 raft_consensus.cc:3058] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:28.911844 6727 raft_consensus.cc:2466] T 1b0ee758121c40bd8590fc46775492b3 P 7470d1d8481a419db3adee1f18ab1f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724bd023063a4d4c97a6f4dfd68a6dce in term 1.
I20250815 07:59:28.911990 6327 raft_consensus.cc:2466] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724bd023063a4d4c97a6f4dfd68a6dce in term 1.
I20250815 07:59:28.912540 6529 leader_election.cc:304] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724bd023063a4d4c97a6f4dfd68a6dce, 7470d1d8481a419db3adee1f18ab1f7d; no voters:
I20250815 07:59:28.912992 6798 raft_consensus.cc:2802] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:59:28.914296 6798 raft_consensus.cc:695] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [term 1 LEADER]: Becoming Leader. State: Replica: 724bd023063a4d4c97a6f4dfd68a6dce, State: Running, Role: LEADER
I20250815 07:59:28.914891 6798 consensus_queue.cc:237] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } }
I20250815 07:59:28.923816 6050 catalog_manager.cc:5582] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce reported cstate change: term changed from 0 to 1, leader changed from <none> to 724bd023063a4d4c97a6f4dfd68a6dce (127.1.121.196). New cstate: current_term: 1 leader_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7470d1d8481a419db3adee1f18ab1f7d" member_type: VOTER last_known_addr { host: "127.1.121.197" port: 40537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "724bd023063a4d4c97a6f4dfd68a6dce" member_type: VOTER last_known_addr { host: "127.1.121.196" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 } health_report { overall_health: UNKNOWN } } }
I20250815 07:59:28.947140 1511 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250815 07:59:28.950258 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 27498835651749758f90684f6cfe6f01 to finish bootstrapping
I20250815 07:59:28.961577 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 724bd023063a4d4c97a6f4dfd68a6dce to finish bootstrapping
I20250815 07:59:28.970566 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7470d1d8481a419db3adee1f18ab1f7d to finish bootstrapping
I20250815 07:59:28.980221 1511 test_util.cc:276] Using random seed: -990712598
W20250815 07:59:28.990347 6373 tablet.cc:2378] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:59:29.001443 1511 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250815 07:59:29.002166 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 6642
W20250815 07:59:29.016855 6639 tablet.cc:2378] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 07:59:29.029840 6529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.197:40537: connect: Connection refused (error 111)
I20250815 07:59:29.030257 6327 raft_consensus.cc:1273] T 1b0ee758121c40bd8590fc46775492b3 P 27498835651749758f90684f6cfe6f01 [term 1 FOLLOWER]: Refusing update from remote peer 724bd023063a4d4c97a6f4dfd68a6dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 07:59:29.032598 6798 consensus_queue.cc:1035] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "27498835651749758f90684f6cfe6f01" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250815 07:59:29.034353 6529 consensus_peers.cc:489] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce -> Peer 7470d1d8481a419db3adee1f18ab1f7d (127.1.121.197:40537): Couldn't send request to peer 7470d1d8481a419db3adee1f18ab1f7d. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.197:40537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250815 07:59:29.048419 6812 mvcc.cc:204] Tried to move back new op lower bound from 7189482573928960000 to 7189482573482754048. Current Snapshot: MvccSnapshot[applied={T|T < 7189482573928960000}]
I20250815 07:59:31.261106 6573 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 07:59:31.288082 6440 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 07:59:31.298957 6307 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 07:59:31.313406 6174 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250815 07:59:31.339365 6529 consensus_peers.cc:489] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce -> Peer 7470d1d8481a419db3adee1f18ab1f7d (127.1.121.197:40537): Couldn't send request to peer 7470d1d8481a419db3adee1f18ab1f7d. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.197:40537: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250815 07:59:32.863672 6307 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 07:59:32.878757 6440 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 07:59:32.889681 6174 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 07:59:32.898604 6573 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250815 07:59:33.912247 6529 consensus_peers.cc:489] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce -> Peer 7470d1d8481a419db3adee1f18ab1f7d (127.1.121.197:40537): Couldn't send request to peer 7470d1d8481a419db3adee1f18ab1f7d. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.197:40537: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250815 07:59:34.486565 6573 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 07:59:34.488113 6307 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 07:59:34.522687 6440 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 07:59:34.530128 6174 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250815 07:59:34.579818 6529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.121.197:40537: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250815 07:59:36.570197 6529 consensus_peers.cc:489] T 1b0ee758121c40bd8590fc46775492b3 P 724bd023063a4d4c97a6f4dfd68a6dce -> Peer 7470d1d8481a419db3adee1f18ab1f7d (127.1.121.197:40537): Couldn't send request to peer 7470d1d8481a419db3adee1f18ab1f7d. Status: Network error: Client connection negotiation failed: client connection to 127.1.121.197:40537: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250815 07:59:36.897745 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 6110
I20250815 07:59:36.918715 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 6243
I20250815 07:59:36.947402 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 6376
I20250815 07:59:36.967619 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 6509
I20250815 07:59:37.001410 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 6017
2025-08-15T07:59:37Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19738 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19738 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250815 07:59:37.055800 1511 test_util.cc:276] Using random seed: -982637024
I20250815 07:59:37.059405 1511 ts_itest-base.cc:115] Starting cluster with:
I20250815 07:59:37.059554 1511 ts_itest-base.cc:116] --------------
I20250815 07:59:37.059686 1511 ts_itest-base.cc:117] 1 tablet servers
I20250815 07:59:37.059803 1511 ts_itest-base.cc:118] 1 replicas per TS
I20250815 07:59:37.059937 1511 ts_itest-base.cc:119] --------------
2025-08-15T07:59:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T07:59:37Z Disabled control of system clock
I20250815 07:59:37.096697 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:42673
--webserver_interface=127.1.121.254
--webserver_port=0
--builtin_ntp_servers=127.1.121.212:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:42673 with env {}
W20250815 07:59:37.387797 6949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:37.388310 6949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:37.388679 6949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:37.414007 6949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:59:37.414216 6949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:37.414394 6949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:59:37.414559 6949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:59:37.443701 6949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:42673
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:42673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:37.444762 6949 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:37.446082 6949 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:37.454430 6955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:38.858996 6954 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 6949
W20250815 07:59:37.455940 6956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:39.146620 6949 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.691s user 0.673s sys 1.011s
W20250815 07:59:39.147886 6949 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.693s user 0.674s sys 1.011s
W20250815 07:59:39.148672 6957 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1692 milliseconds
W20250815 07:59:39.148882 6958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:39.149143 6949 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:39.153040 6949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:39.155352 6949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:39.156700 6949 hybrid_clock.cc:648] HybridClock initialized: now 1755244779156658 us; error 37 us; skew 500 ppm
I20250815 07:59:39.157454 6949 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:39.163187 6949 webserver.cc:480] Webserver started at http://127.1.121.254:33861/ using document root <none> and password file <none>
I20250815 07:59:39.164103 6949 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:39.164314 6949 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:39.164747 6949 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:39.169020 6949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3e31f743285f4725bac1e204175fc20b"
format_stamp: "Formatted at 2025-08-15 07:59:39 on dist-test-slave-dlgj"
I20250815 07:59:39.170111 6949 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3e31f743285f4725bac1e204175fc20b"
format_stamp: "Formatted at 2025-08-15 07:59:39 on dist-test-slave-dlgj"
I20250815 07:59:39.176932 6949 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250815 07:59:39.182102 6965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:39.183012 6949 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.003s
I20250815 07:59:39.183310 6949 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3e31f743285f4725bac1e204175fc20b"
format_stamp: "Formatted at 2025-08-15 07:59:39 on dist-test-slave-dlgj"
I20250815 07:59:39.183626 6949 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:39.235203 6949 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:39.236445 6949 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:39.236810 6949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:39.298683 6949 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:42673
I20250815 07:59:39.298758 7016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:42673 every 8 connection(s)
I20250815 07:59:39.301051 6949 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250815 07:59:39.302551 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 6949
I20250815 07:59:39.303066 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250815 07:59:39.306468 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:39.328444 7017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b: Bootstrap starting.
I20250815 07:59:39.333695 7017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:39.335043 7017 log.cc:826] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:39.339195 7017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b: No bootstrap required, opened a new log
I20250815 07:59:39.354849 7017 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e31f743285f4725bac1e204175fc20b" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 42673 } }
I20250815 07:59:39.355391 7017 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:39.355643 7017 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e31f743285f4725bac1e204175fc20b, State: Initialized, Role: FOLLOWER
I20250815 07:59:39.356298 7017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e31f743285f4725bac1e204175fc20b" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 42673 } }
I20250815 07:59:39.356914 7017 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:59:39.357244 7017 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:59:39.357662 7017 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:39.361370 7017 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e31f743285f4725bac1e204175fc20b" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 42673 } }
I20250815 07:59:39.361974 7017 leader_election.cc:304] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e31f743285f4725bac1e204175fc20b; no voters:
I20250815 07:59:39.363533 7017 leader_election.cc:290] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:59:39.364084 7022 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:59:39.366525 7022 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [term 1 LEADER]: Becoming Leader. State: Replica: 3e31f743285f4725bac1e204175fc20b, State: Running, Role: LEADER
I20250815 07:59:39.367120 7022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e31f743285f4725bac1e204175fc20b" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 42673 } }
I20250815 07:59:39.367862 7017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:59:39.375120 7024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e31f743285f4725bac1e204175fc20b. Latest consensus state: current_term: 1 leader_uuid: "3e31f743285f4725bac1e204175fc20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e31f743285f4725bac1e204175fc20b" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 42673 } } }
I20250815 07:59:39.375537 7023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e31f743285f4725bac1e204175fc20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e31f743285f4725bac1e204175fc20b" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 42673 } } }
I20250815 07:59:39.375830 7024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [sys.catalog]: This master's current role is: LEADER
I20250815 07:59:39.376045 7023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b [sys.catalog]: This master's current role is: LEADER
I20250815 07:59:39.378348 7029 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:59:39.390833 7029 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:59:39.406065 7029 catalog_manager.cc:1349] Generated new cluster ID: e1313c37e1b54ea0873c0dd32f338d41
I20250815 07:59:39.406332 7029 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:59:39.417634 7029 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 07:59:39.418840 7029 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:59:39.437136 7029 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3e31f743285f4725bac1e204175fc20b: Generated new TSK 0
I20250815 07:59:39.437939 7029 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:59:39.445000 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:0
--local_ip_for_outbound_sockets=127.1.121.193
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:42673
--builtin_ntp_servers=127.1.121.212:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250815 07:59:39.734997 7041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:39.735529 7041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:39.736163 7041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:39.764039 7041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:39.764811 7041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:59:39.796530 7041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:42673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:39.797881 7041 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:39.799412 7041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:39.810730 7048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:39.812101 7049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:41.167608 7051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:41.169737 7050 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1357 milliseconds
W20250815 07:59:41.170789 7041 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.359s user 0.436s sys 0.918s
W20250815 07:59:41.171116 7041 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.359s user 0.436s sys 0.918s
I20250815 07:59:41.171414 7041 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:41.172940 7041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:41.175978 7041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:41.177475 7041 hybrid_clock.cc:648] HybridClock initialized: now 1755244781177411 us; error 75 us; skew 500 ppm
I20250815 07:59:41.178613 7041 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:41.188717 7041 webserver.cc:480] Webserver started at http://127.1.121.193:36001/ using document root <none> and password file <none>
I20250815 07:59:41.190025 7041 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:41.190302 7041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:41.190940 7041 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:41.197810 7041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2b558eade4a045b893f49db28f5a9e09"
format_stamp: "Formatted at 2025-08-15 07:59:41 on dist-test-slave-dlgj"
I20250815 07:59:41.199282 7041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2b558eade4a045b893f49db28f5a9e09"
format_stamp: "Formatted at 2025-08-15 07:59:41 on dist-test-slave-dlgj"
I20250815 07:59:41.210664 7041 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.008s sys 0.004s
I20250815 07:59:41.218327 7058 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:41.220914 7041 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20250815 07:59:41.221540 7041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2b558eade4a045b893f49db28f5a9e09"
format_stamp: "Formatted at 2025-08-15 07:59:41 on dist-test-slave-dlgj"
I20250815 07:59:41.222193 7041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:41.300619 7041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:41.302479 7041 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:41.302958 7041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:41.306063 7041 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:41.311164 7041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:41.311411 7041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:41.311676 7041 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:41.311870 7041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:41.458091 7041 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:33241
I20250815 07:59:41.458155 7170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:33241 every 8 connection(s)
I20250815 07:59:41.461578 7041 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250815 07:59:41.470938 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 7041
I20250815 07:59:41.471697 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755244639110353-1511-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250815 07:59:41.484303 7171 heartbeater.cc:344] Connected to a master server at 127.1.121.254:42673
I20250815 07:59:41.484663 7171 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:41.485507 7171 heartbeater.cc:507] Master 127.1.121.254:42673 requested a full tablet report, sending...
I20250815 07:59:41.487637 6982 ts_manager.cc:194] Registered new tserver with Master: 2b558eade4a045b893f49db28f5a9e09 (127.1.121.193:33241)
I20250815 07:59:41.489786 6982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:38459
I20250815 07:59:41.491828 1511 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250815 07:59:41.523881 6982 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50846:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250815 07:59:41.582252 7106 tablet_service.cc:1468] Processing CreateTablet for tablet c5ce43bbf6204f6cbd7d2cc6c420673a (DEFAULT_TABLE table=TestTable [id=cdd972a81a68418b928fdeabc72e48c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:59:41.583616 7106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5ce43bbf6204f6cbd7d2cc6c420673a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:41.601353 7186 tablet_bootstrap.cc:492] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09: Bootstrap starting.
I20250815 07:59:41.605973 7186 tablet_bootstrap.cc:654] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:41.607453 7186 log.cc:826] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:41.611672 7186 tablet_bootstrap.cc:492] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09: No bootstrap required, opened a new log
I20250815 07:59:41.611984 7186 ts_tablet_manager.cc:1397] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250815 07:59:41.626595 7186 raft_consensus.cc:357] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b558eade4a045b893f49db28f5a9e09" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 33241 } }
I20250815 07:59:41.627017 7186 raft_consensus.cc:383] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:41.627220 7186 raft_consensus.cc:738] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b558eade4a045b893f49db28f5a9e09, State: Initialized, Role: FOLLOWER
I20250815 07:59:41.627857 7186 consensus_queue.cc:260] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b558eade4a045b893f49db28f5a9e09" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 33241 } }
I20250815 07:59:41.628358 7186 raft_consensus.cc:397] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:59:41.628614 7186 raft_consensus.cc:491] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:59:41.628907 7186 raft_consensus.cc:3058] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:41.632678 7186 raft_consensus.cc:513] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b558eade4a045b893f49db28f5a9e09" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 33241 } }
I20250815 07:59:41.633318 7186 leader_election.cc:304] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b558eade4a045b893f49db28f5a9e09; no voters:
I20250815 07:59:41.635392 7186 leader_election.cc:290] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:59:41.635742 7188 raft_consensus.cc:2802] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:59:41.638657 7171 heartbeater.cc:499] Master 127.1.121.254:42673 was elected leader, sending a full tablet report...
I20250815 07:59:41.639636 7186 ts_tablet_manager.cc:1428] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09: Time spent starting tablet: real 0.027s user 0.015s sys 0.011s
I20250815 07:59:41.640213 7188 raft_consensus.cc:695] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [term 1 LEADER]: Becoming Leader. State: Replica: 2b558eade4a045b893f49db28f5a9e09, State: Running, Role: LEADER
I20250815 07:59:41.640861 7188 consensus_queue.cc:237] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b558eade4a045b893f49db28f5a9e09" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 33241 } }
I20250815 07:59:41.650104 6982 catalog_manager.cc:5582] T c5ce43bbf6204f6cbd7d2cc6c420673a P 2b558eade4a045b893f49db28f5a9e09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b558eade4a045b893f49db28f5a9e09 (127.1.121.193). New cstate: current_term: 1 leader_uuid: "2b558eade4a045b893f49db28f5a9e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b558eade4a045b893f49db28f5a9e09" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 33241 } health_report { overall_health: HEALTHY } } }
I20250815 07:59:41.678495 1511 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250815 07:59:41.681344 1511 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2b558eade4a045b893f49db28f5a9e09 to finish bootstrapping
I20250815 07:59:44.242115 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 7041
I20250815 07:59:44.266331 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 6949
2025-08-15T07:59:44Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7259 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7259 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250815 07:59:44.315493 1511 test_util.cc:276] Using random seed: -975377327
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (10 ms)
[----------] 1 test from ListTableCliParamTest (10 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-15T07:59:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-15T07:59:44Z Disabled control of system clock
I20250815 07:59:44.360160 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:33557
--webserver_interface=127.1.121.254
--webserver_port=0
--builtin_ntp_servers=127.1.121.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:33557 with env {}
W20250815 07:59:44.624054 7214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:44.624583 7214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:44.624969 7214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:44.652911 7214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 07:59:44.653187 7214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:44.653415 7214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 07:59:44.653635 7214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 07:59:44.689179 7214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:33557
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:44.690368 7214 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:44.691814 7214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:44.701275 7220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:44.701833 7221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:45.770140 7222 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1066 milliseconds
W20250815 07:59:45.771597 7223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:45.773272 7214 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.072s user 0.289s sys 0.775s
W20250815 07:59:45.773504 7214 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.072s user 0.289s sys 0.775s
I20250815 07:59:45.773726 7214 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:45.774729 7214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:45.777043 7214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:45.778374 7214 hybrid_clock.cc:648] HybridClock initialized: now 1755244785778338 us; error 31 us; skew 500 ppm
I20250815 07:59:45.779156 7214 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:45.785351 7214 webserver.cc:480] Webserver started at http://127.1.121.254:40927/ using document root <none> and password file <none>
I20250815 07:59:45.786211 7214 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:45.786443 7214 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:45.786859 7214 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:45.790760 7214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/instance:
uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4"
format_stamp: "Formatted at 2025-08-15 07:59:45 on dist-test-slave-dlgj"
I20250815 07:59:45.791688 7214 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal/instance:
uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4"
format_stamp: "Formatted at 2025-08-15 07:59:45 on dist-test-slave-dlgj"
I20250815 07:59:45.798471 7214 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20250815 07:59:45.803694 7230 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:45.804662 7214 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250815 07:59:45.804924 7214 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4"
format_stamp: "Formatted at 2025-08-15 07:59:45 on dist-test-slave-dlgj"
I20250815 07:59:45.805200 7214 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:45.875916 7214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:45.877178 7214 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:45.877558 7214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:45.938876 7214 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:33557
I20250815 07:59:45.938959 7281 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:33557 every 8 connection(s)
I20250815 07:59:45.941344 7214 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/info.pb
I20250815 07:59:45.945770 7282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:45.949386 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 7214
I20250815 07:59:45.949774 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal/instance
I20250815 07:59:45.966675 7282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4: Bootstrap starting.
I20250815 07:59:45.971262 7282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:45.972710 7282 log.cc:826] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:45.976233 7282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4: No bootstrap required, opened a new log
I20250815 07:59:45.991824 7282 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } }
I20250815 07:59:45.992416 7282 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:45.992611 7282 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e63178f5a7949a2a59f6abd7f3bc3c4, State: Initialized, Role: FOLLOWER
I20250815 07:59:45.993093 7282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } }
I20250815 07:59:45.993484 7282 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:59:45.993669 7282 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:59:45.993898 7282 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:45.997396 7282 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } }
I20250815 07:59:45.997895 7282 leader_election.cc:304] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e63178f5a7949a2a59f6abd7f3bc3c4; no voters:
I20250815 07:59:45.999406 7282 leader_election.cc:290] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:59:46.000131 7287 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:59:46.002143 7287 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [term 1 LEADER]: Becoming Leader. State: Replica: 3e63178f5a7949a2a59f6abd7f3bc3c4, State: Running, Role: LEADER
I20250815 07:59:46.002748 7287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } }
I20250815 07:59:46.003593 7282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [sys.catalog]: configured and running, proceeding with master startup.
I20250815 07:59:46.009368 7288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } } }
I20250815 07:59:46.009310 7289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e63178f5a7949a2a59f6abd7f3bc3c4. Latest consensus state: current_term: 1 leader_uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e63178f5a7949a2a59f6abd7f3bc3c4" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } } }
I20250815 07:59:46.009902 7289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [sys.catalog]: This master's current role is: LEADER
I20250815 07:59:46.009902 7288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4 [sys.catalog]: This master's current role is: LEADER
I20250815 07:59:46.013128 7294 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 07:59:46.024099 7294 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 07:59:46.038359 7294 catalog_manager.cc:1349] Generated new cluster ID: 407640795571457f9a2292bc746636b2
I20250815 07:59:46.038612 7294 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 07:59:46.076491 7294 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 07:59:46.077633 7294 catalog_manager.cc:1506] Loading token signing keys...
I20250815 07:59:46.090421 7294 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3e63178f5a7949a2a59f6abd7f3bc3c4: Generated new TSK 0
I20250815 07:59:46.091179 7294 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250815 07:59:46.103408 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:0
--local_ip_for_outbound_sockets=127.1.121.193
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:33557
--builtin_ntp_servers=127.1.121.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250815 07:59:46.383168 7306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:46.383709 7306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:46.384212 7306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:46.410580 7306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:46.411309 7306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 07:59:46.441255 7306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=0
--tserver_master_addrs=127.1.121.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:46.442376 7306 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:46.443729 7306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:46.454308 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:46.455524 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:47.571709 7315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:47.573740 7306 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.117s user 0.000s sys 0.006s
W20250815 07:59:47.574082 7306 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.118s user 0.000s sys 0.006s
I20250815 07:59:47.574358 7306 server_base.cc:1047] running on GCE node
I20250815 07:59:47.575702 7306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:47.580549 7306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:47.582039 7306 hybrid_clock.cc:648] HybridClock initialized: now 1755244787581966 us; error 65 us; skew 500 ppm
I20250815 07:59:47.583079 7306 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:47.592087 7306 webserver.cc:480] Webserver started at http://127.1.121.193:32819/ using document root <none> and password file <none>
I20250815 07:59:47.593425 7306 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:47.593729 7306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:47.594307 7306 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:47.601261 7306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/instance:
uuid: "283bb3462b894e23a649bc318b5011bb"
format_stamp: "Formatted at 2025-08-15 07:59:47 on dist-test-slave-dlgj"
I20250815 07:59:47.602773 7306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal/instance:
uuid: "283bb3462b894e23a649bc318b5011bb"
format_stamp: "Formatted at 2025-08-15 07:59:47 on dist-test-slave-dlgj"
I20250815 07:59:47.612491 7306 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20250815 07:59:47.620028 7322 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:47.621304 7306 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250815 07:59:47.621706 7306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
uuid: "283bb3462b894e23a649bc318b5011bb"
format_stamp: "Formatted at 2025-08-15 07:59:47 on dist-test-slave-dlgj"
I20250815 07:59:47.622118 7306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:47.704931 7306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:47.706964 7306 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:47.707525 7306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:47.710582 7306 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:47.714394 7306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:47.714570 7306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:47.714792 7306 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:47.714941 7306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:47.866653 7306 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:35817
I20250815 07:59:47.866806 7434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:35817 every 8 connection(s)
I20250815 07:59:47.869017 7306 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/info.pb
I20250815 07:59:47.879734 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 7306
I20250815 07:59:47.880223 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal/instance
I20250815 07:59:47.887338 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:0
--local_ip_for_outbound_sockets=127.1.121.194
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:33557
--builtin_ntp_servers=127.1.121.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 07:59:47.889564 7435 heartbeater.cc:344] Connected to a master server at 127.1.121.254:33557
I20250815 07:59:47.889920 7435 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:47.890763 7435 heartbeater.cc:507] Master 127.1.121.254:33557 requested a full tablet report, sending...
I20250815 07:59:47.892799 7247 ts_manager.cc:194] Registered new tserver with Master: 283bb3462b894e23a649bc318b5011bb (127.1.121.193:35817)
I20250815 07:59:47.894490 7247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:52667
W20250815 07:59:48.174401 7439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:48.174906 7439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:48.175403 7439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:48.203953 7439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:48.204820 7439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 07:59:48.235817 7439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=0
--tserver_master_addrs=127.1.121.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:48.236965 7439 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:48.238348 7439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:48.249013 7445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:48.897511 7435 heartbeater.cc:499] Master 127.1.121.254:33557 was elected leader, sending a full tablet report...
W20250815 07:59:49.652756 7444 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7439
W20250815 07:59:49.814495 7444 kernel_stack_watchdog.cc:198] Thread 7439 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250815 07:59:48.249763 7446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:49.815178 7439 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.567s user 0.519s sys 1.043s
W20250815 07:59:49.815582 7439 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.567s user 0.519s sys 1.043s
W20250815 07:59:49.817312 7448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:49.819578 7447 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1566 milliseconds
I20250815 07:59:49.819584 7439 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:49.820691 7439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:49.822517 7439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:49.823828 7439 hybrid_clock.cc:648] HybridClock initialized: now 1755244789823771 us; error 55 us; skew 500 ppm
I20250815 07:59:49.824534 7439 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:49.830237 7439 webserver.cc:480] Webserver started at http://127.1.121.194:35733/ using document root <none> and password file <none>
I20250815 07:59:49.831073 7439 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:49.831257 7439 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:49.831700 7439 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:49.837314 7439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/instance:
uuid: "5f38cf428cd34649b8c24f8930910161"
format_stamp: "Formatted at 2025-08-15 07:59:49 on dist-test-slave-dlgj"
I20250815 07:59:49.838424 7439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal/instance:
uuid: "5f38cf428cd34649b8c24f8930910161"
format_stamp: "Formatted at 2025-08-15 07:59:49 on dist-test-slave-dlgj"
I20250815 07:59:49.845134 7439 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250815 07:59:49.850239 7456 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:49.851094 7439 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250815 07:59:49.851367 7439 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
uuid: "5f38cf428cd34649b8c24f8930910161"
format_stamp: "Formatted at 2025-08-15 07:59:49 on dist-test-slave-dlgj"
I20250815 07:59:49.851644 7439 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:49.894809 7439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:49.896193 7439 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:49.896580 7439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:49.898844 7439 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:49.902509 7439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:49.902686 7439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:49.902901 7439 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:49.903038 7439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:50.024678 7439 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:34441
I20250815 07:59:50.024781 7568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:34441 every 8 connection(s)
I20250815 07:59:50.026911 7439 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/info.pb
I20250815 07:59:50.029210 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 7439
I20250815 07:59:50.030047 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal/instance
I20250815 07:59:50.037740 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:0
--local_ip_for_outbound_sockets=127.1.121.195
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:33557
--builtin_ntp_servers=127.1.121.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 07:59:50.050083 7569 heartbeater.cc:344] Connected to a master server at 127.1.121.254:33557
I20250815 07:59:50.050484 7569 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:50.051461 7569 heartbeater.cc:507] Master 127.1.121.254:33557 requested a full tablet report, sending...
I20250815 07:59:50.053426 7247 ts_manager.cc:194] Registered new tserver with Master: 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441)
I20250815 07:59:50.054513 7247 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:59269
W20250815 07:59:50.315303 7573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:50.315809 7573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 07:59:50.316309 7573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:50.344995 7573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 07:59:50.345850 7573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 07:59:50.377377 7573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=0
--tserver_master_addrs=127.1.121.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 07:59:50.378656 7573 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:50.380261 7573 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:50.390743 7579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:51.057351 7569 heartbeater.cc:499] Master 127.1.121.254:33557 was elected leader, sending a full tablet report...
W20250815 07:59:50.392663 7580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:51.450542 7582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:51.452275 7581 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1056 milliseconds
I20250815 07:59:51.452396 7573 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 07:59:51.453404 7573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 07:59:51.455695 7573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 07:59:51.457062 7573 hybrid_clock.cc:648] HybridClock initialized: now 1755244791457020 us; error 43 us; skew 500 ppm
I20250815 07:59:51.457753 7573 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:51.463649 7573 webserver.cc:480] Webserver started at http://127.1.121.195:38635/ using document root <none> and password file <none>
I20250815 07:59:51.464587 7573 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:51.464776 7573 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:51.465221 7573 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:51.469091 7573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/instance:
uuid: "9abb03c51da8480ea4190a3d5fcbf1d0"
format_stamp: "Formatted at 2025-08-15 07:59:51 on dist-test-slave-dlgj"
I20250815 07:59:51.470088 7573 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal/instance:
uuid: "9abb03c51da8480ea4190a3d5fcbf1d0"
format_stamp: "Formatted at 2025-08-15 07:59:51 on dist-test-slave-dlgj"
I20250815 07:59:51.475950 7573 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250815 07:59:51.480830 7589 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:51.481907 7573 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250815 07:59:51.482159 7573 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
uuid: "9abb03c51da8480ea4190a3d5fcbf1d0"
format_stamp: "Formatted at 2025-08-15 07:59:51 on dist-test-slave-dlgj"
I20250815 07:59:51.482410 7573 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:51.536885 7573 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:51.538231 7573 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:51.538650 7573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:51.541057 7573 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 07:59:51.544704 7573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250815 07:59:51.544876 7573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:51.545110 7573 ts_tablet_manager.cc:610] Registered 0 tablets
I20250815 07:59:51.545253 7573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:51.666671 7573 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:42845
I20250815 07:59:51.666761 7701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:42845 every 8 connection(s)
I20250815 07:59:51.669118 7573 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/info.pb
I20250815 07:59:51.675235 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 7573
I20250815 07:59:51.675630 1511 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal/instance
I20250815 07:59:51.692709 7702 heartbeater.cc:344] Connected to a master server at 127.1.121.254:33557
I20250815 07:59:51.693137 7702 heartbeater.cc:461] Registering TS with master...
I20250815 07:59:51.694097 7702 heartbeater.cc:507] Master 127.1.121.254:33557 requested a full tablet report, sending...
I20250815 07:59:51.696141 7246 ts_manager.cc:194] Registered new tserver with Master: 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845)
I20250815 07:59:51.697465 7246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:57435
I20250815 07:59:51.707877 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 07:59:51.736243 1511 test_util.cc:276] Using random seed: -967956581
I20250815 07:59:51.772779 7246 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45258:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250815 07:59:51.774932 7246 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 07:59:51.821534 7637 tablet_service.cc:1468] Processing CreateTablet for tablet 5090f1bb5ffb42c09350308bbcef6940 (DEFAULT_TABLE table=pre_rebuild [id=33b3f7f0248949a1b6dbad77e5427823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:59:51.823184 7637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5090f1bb5ffb42c09350308bbcef6940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:51.829617 7504 tablet_service.cc:1468] Processing CreateTablet for tablet 5090f1bb5ffb42c09350308bbcef6940 (DEFAULT_TABLE table=pre_rebuild [id=33b3f7f0248949a1b6dbad77e5427823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:59:51.831867 7504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5090f1bb5ffb42c09350308bbcef6940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:51.832398 7370 tablet_service.cc:1468] Processing CreateTablet for tablet 5090f1bb5ffb42c09350308bbcef6940 (DEFAULT_TABLE table=pre_rebuild [id=33b3f7f0248949a1b6dbad77e5427823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 07:59:51.834143 7370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5090f1bb5ffb42c09350308bbcef6940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:51.849691 7726 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Bootstrap starting.
I20250815 07:59:51.852651 7727 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Bootstrap starting.
I20250815 07:59:51.856868 7726 tablet_bootstrap.cc:654] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:51.858575 7726 log.cc:826] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:51.859561 7727 tablet_bootstrap.cc:654] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:51.860168 7728 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Bootstrap starting.
I20250815 07:59:51.862174 7727 log.cc:826] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:51.864493 7726 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: No bootstrap required, opened a new log
I20250815 07:59:51.865069 7726 ts_tablet_manager.cc:1397] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20250815 07:59:51.868515 7728 tablet_bootstrap.cc:654] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:51.870941 7728 log.cc:826] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:51.872960 7727 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: No bootstrap required, opened a new log
I20250815 07:59:51.873464 7727 ts_tablet_manager.cc:1397] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.011s
I20250815 07:59:51.886726 7728 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: No bootstrap required, opened a new log
I20250815 07:59:51.887279 7728 ts_tablet_manager.cc:1397] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.014s
I20250815 07:59:51.894533 7726 raft_consensus.cc:357] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 07:59:51.895453 7726 raft_consensus.cc:383] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:51.895771 7726 raft_consensus.cc:738] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb03c51da8480ea4190a3d5fcbf1d0, State: Initialized, Role: FOLLOWER
I20250815 07:59:51.896716 7726 consensus_queue.cc:260] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 07:59:51.898262 7727 raft_consensus.cc:357] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 07:59:51.899185 7727 raft_consensus.cc:383] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:51.899529 7727 raft_consensus.cc:738] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f38cf428cd34649b8c24f8930910161, State: Initialized, Role: FOLLOWER
I20250815 07:59:51.900585 7727 consensus_queue.cc:260] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 07:59:51.905577 7702 heartbeater.cc:499] Master 127.1.121.254:33557 was elected leader, sending a full tablet report...
I20250815 07:59:51.908377 7727 ts_tablet_manager.cc:1428] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Time spent starting tablet: real 0.035s user 0.024s sys 0.008s
I20250815 07:59:51.910602 7726 ts_tablet_manager.cc:1428] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Time spent starting tablet: real 0.045s user 0.026s sys 0.011s
I20250815 07:59:51.913203 7732 raft_consensus.cc:491] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 07:59:51.913816 7732 raft_consensus.cc:513] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 07:59:51.917865 7728 raft_consensus.cc:357] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 07:59:51.920661 7728 raft_consensus.cc:383] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:51.921262 7732 leader_election.cc:290] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283bb3462b894e23a649bc318b5011bb (127.1.121.193:35817), 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441)
I20250815 07:59:51.920984 7728 raft_consensus.cc:738] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283bb3462b894e23a649bc318b5011bb, State: Initialized, Role: FOLLOWER
I20250815 07:59:51.923550 7728 consensus_queue.cc:260] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
W20250815 07:59:51.923617 7703 tablet.cc:2378] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:59:51.927501 7728 ts_tablet_manager.cc:1428] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Time spent starting tablet: real 0.040s user 0.031s sys 0.000s
I20250815 07:59:51.931903 7390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283bb3462b894e23a649bc318b5011bb" is_pre_election: true
I20250815 07:59:51.932720 7390 raft_consensus.cc:2466] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9abb03c51da8480ea4190a3d5fcbf1d0 in term 0.
I20250815 07:59:51.934233 7591 leader_election.cc:304] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283bb3462b894e23a649bc318b5011bb, 9abb03c51da8480ea4190a3d5fcbf1d0; no voters:
I20250815 07:59:51.935048 7732 raft_consensus.cc:2802] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 07:59:51.935410 7732 raft_consensus.cc:491] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 07:59:51.935355 7524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f38cf428cd34649b8c24f8930910161" is_pre_election: true
I20250815 07:59:51.935739 7732 raft_consensus.cc:3058] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:51.936236 7524 raft_consensus.cc:2466] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9abb03c51da8480ea4190a3d5fcbf1d0 in term 0.
I20250815 07:59:51.941138 7732 raft_consensus.cc:513] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 07:59:51.942473 7732 leader_election.cc:290] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [CANDIDATE]: Term 1 election: Requested vote from peers 283bb3462b894e23a649bc318b5011bb (127.1.121.193:35817), 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441)
I20250815 07:59:51.943146 7390 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283bb3462b894e23a649bc318b5011bb"
I20250815 07:59:51.943421 7524 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f38cf428cd34649b8c24f8930910161"
I20250815 07:59:51.943663 7390 raft_consensus.cc:3058] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:51.943899 7524 raft_consensus.cc:3058] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:51.950520 7390 raft_consensus.cc:2466] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9abb03c51da8480ea4190a3d5fcbf1d0 in term 1.
I20250815 07:59:51.950608 7524 raft_consensus.cc:2466] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9abb03c51da8480ea4190a3d5fcbf1d0 in term 1.
I20250815 07:59:51.951576 7592 leader_election.cc:304] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f38cf428cd34649b8c24f8930910161, 9abb03c51da8480ea4190a3d5fcbf1d0; no voters:
I20250815 07:59:51.952194 7732 raft_consensus.cc:2802] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:59:51.953794 7732 raft_consensus.cc:695] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 LEADER]: Becoming Leader. State: Replica: 9abb03c51da8480ea4190a3d5fcbf1d0, State: Running, Role: LEADER
I20250815 07:59:51.954649 7732 consensus_queue.cc:237] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 07:59:51.966069 7246 catalog_manager.cc:5582] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195). New cstate: current_term: 1 leader_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } health_report { overall_health: UNKNOWN } } }
W20250815 07:59:52.033387 7570 tablet.cc:2378] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250815 07:59:52.127616 7436 tablet.cc:2378] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 07:59:52.135941 7390 raft_consensus.cc:1273] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Refusing update from remote peer 9abb03c51da8480ea4190a3d5fcbf1d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 07:59:52.135941 7524 raft_consensus.cc:1273] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Refusing update from remote peer 9abb03c51da8480ea4190a3d5fcbf1d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 07:59:52.137595 7732 consensus_queue.cc:1035] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250815 07:59:52.138610 7736 consensus_queue.cc:1035] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 07:59:52.163928 7743 mvcc.cc:204] Tried to move back new op lower bound from 7189482668571852800 to 7189482667853123584. Current Snapshot: MvccSnapshot[applied={T|T < 7189482668571852800}]
I20250815 07:59:52.195482 7746 mvcc.cc:204] Tried to move back new op lower bound from 7189482668571852800 to 7189482667853123584. Current Snapshot: MvccSnapshot[applied={T|T < 7189482668571852800}]
I20250815 07:59:56.899480 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 7214
W20250815 07:59:57.263113 7702 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:33557 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:33557: connect: Connection refused (error 111)
W20250815 07:59:57.265516 7435 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:33557 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:33557: connect: Connection refused (error 111)
W20250815 07:59:57.294423 7775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 07:59:57.294943 7775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 07:59:57.306007 7569 heartbeater.cc:646] Failed to heartbeat to 127.1.121.254:33557 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.121.254:33557: connect: Connection refused (error 111)
W20250815 07:59:57.323509 7775 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250815 07:59:58.660892 7775 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.302s user 0.385s sys 0.901s
W20250815 07:59:58.661195 7775 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.302s user 0.385s sys 0.901s
I20250815 07:59:58.742141 7775 minidump.cc:252] Setting minidump size limit to 20M
I20250815 07:59:58.744544 7775 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 07:59:58.745580 7775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 07:59:58.755683 7810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:58.757534 7811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 07:59:58.829684 7813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 07:59:58.829751 7775 server_base.cc:1047] running on GCE node
I20250815 07:59:58.830925 7775 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250815 07:59:58.831332 7775 hybrid_clock.cc:648] HybridClock initialized: now 1755244798831309 us; error 69296 us; skew 500 ppm
I20250815 07:59:58.831955 7775 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 07:59:58.836104 7775 webserver.cc:480] Webserver started at http://0.0.0.0:43607/ using document root <none> and password file <none>
I20250815 07:59:58.836903 7775 fs_manager.cc:362] Metadata directory not provided
I20250815 07:59:58.837105 7775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 07:59:58.837497 7775 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250815 07:59:58.841554 7775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/instance:
uuid: "8f1319d17be64fcc96f00fdfcb0adb3a"
format_stamp: "Formatted at 2025-08-15 07:59:58 on dist-test-slave-dlgj"
I20250815 07:59:58.842609 7775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal/instance:
uuid: "8f1319d17be64fcc96f00fdfcb0adb3a"
format_stamp: "Formatted at 2025-08-15 07:59:58 on dist-test-slave-dlgj"
I20250815 07:59:58.847854 7775 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20250815 07:59:58.852065 7818 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 07:59:58.852790 7775 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20250815 07:59:58.853075 7775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
uuid: "8f1319d17be64fcc96f00fdfcb0adb3a"
format_stamp: "Formatted at 2025-08-15 07:59:58 on dist-test-slave-dlgj"
I20250815 07:59:58.853363 7775 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 07:59:58.917945 7775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 07:59:58.919315 7775 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 07:59:58.919754 7775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 07:59:58.924190 7775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 07:59:58.937737 7775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Bootstrap starting.
I20250815 07:59:58.942417 7775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Neither blocks nor log segments found. Creating new log.
I20250815 07:59:58.944003 7775 log.cc:826] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Log is configured to *not* fsync() on all Append() calls
I20250815 07:59:58.947556 7775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: No bootstrap required, opened a new log
I20250815 07:59:58.962313 7775 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER }
I20250815 07:59:58.962805 7775 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 07:59:58.963022 7775 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1319d17be64fcc96f00fdfcb0adb3a, State: Initialized, Role: FOLLOWER
I20250815 07:59:58.963625 7775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER }
I20250815 07:59:58.964087 7775 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 07:59:58.964355 7775 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 07:59:58.964658 7775 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 0 FOLLOWER]: Advancing to term 1
I20250815 07:59:58.968320 7775 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER }
I20250815 07:59:58.968943 7775 leader_election.cc:304] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f1319d17be64fcc96f00fdfcb0adb3a; no voters:
I20250815 07:59:58.970599 7775 leader_election.cc:290] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250815 07:59:58.970813 7825 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 FOLLOWER]: Leader election won for term 1
I20250815 07:59:58.972743 7825 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 LEADER]: Becoming Leader. State: Replica: 8f1319d17be64fcc96f00fdfcb0adb3a, State: Running, Role: LEADER
I20250815 07:59:58.973503 7825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER }
I20250815 07:59:58.980208 7826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER } }
I20250815 07:59:58.980190 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f1319d17be64fcc96f00fdfcb0adb3a. Latest consensus state: current_term: 1 leader_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER } }
I20250815 07:59:58.980916 7826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: This master's current role is: LEADER
I20250815 07:59:58.980940 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: This master's current role is: LEADER
I20250815 07:59:58.991325 7775 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: stopping tablet replica
I20250815 07:59:58.991870 7775 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 LEADER]: Raft consensus shutting down.
I20250815 07:59:58.992285 7775 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250815 07:59:58.994067 7775 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250815 07:59:58.994438 7775 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250815 07:59:59.019570 7775 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250815 08:00:00.048213 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 7306
I20250815 08:00:00.079615 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 7439
I20250815 08:00:00.109120 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 7573
I20250815 08:00:00.152818 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:33557
--webserver_interface=127.1.121.254
--webserver_port=40927
--builtin_ntp_servers=127.1.121.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.121.254:33557 with env {}
W20250815 08:00:00.426106 7835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 08:00:00.426661 7835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 08:00:00.427093 7835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 08:00:00.453910 7835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250815 08:00:00.454197 7835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 08:00:00.454428 7835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250815 08:00:00.454648 7835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250815 08:00:00.485492 7835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.121.254:33557
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.121.254:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.121.254
--webserver_port=40927
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 08:00:00.486708 7835 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 08:00:00.488188 7835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 08:00:00.497534 7841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 08:00:00.498081 7842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 08:00:01.595052 7844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 08:00:01.598363 7843 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1098 milliseconds
W20250815 08:00:01.599243 7835 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.102s user 0.322s sys 0.777s
W20250815 08:00:01.599514 7835 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.102s user 0.322s sys 0.777s
I20250815 08:00:01.599741 7835 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 08:00:01.600864 7835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 08:00:01.603396 7835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 08:00:01.604748 7835 hybrid_clock.cc:648] HybridClock initialized: now 1755244801604697 us; error 61 us; skew 500 ppm
I20250815 08:00:01.605516 7835 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 08:00:01.612470 7835 webserver.cc:480] Webserver started at http://127.1.121.254:40927/ using document root <none> and password file <none>
I20250815 08:00:01.613453 7835 fs_manager.cc:362] Metadata directory not provided
I20250815 08:00:01.613708 7835 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 08:00:01.621982 7835 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20250815 08:00:01.626508 7851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 08:00:01.627724 7835 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 08:00:01.628121 7835 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
uuid: "8f1319d17be64fcc96f00fdfcb0adb3a"
format_stamp: "Formatted at 2025-08-15 07:59:58 on dist-test-slave-dlgj"
I20250815 08:00:01.630106 7835 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 08:00:01.704672 7835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 08:00:01.706058 7835 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 08:00:01.706451 7835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 08:00:01.772493 7835 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.254:33557
I20250815 08:00:01.772568 7902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.254:33557 every 8 connection(s)
I20250815 08:00:01.775061 7835 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/info.pb
I20250815 08:00:01.779693 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 7835
I20250815 08:00:01.781212 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.193:35817
--local_ip_for_outbound_sockets=127.1.121.193
--tserver_master_addrs=127.1.121.254:33557
--webserver_port=32819
--webserver_interface=127.1.121.193
--builtin_ntp_servers=127.1.121.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 08:00:01.784773 7903 sys_catalog.cc:263] Verifying existing consensus state
I20250815 08:00:01.800403 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Bootstrap starting.
I20250815 08:00:01.814076 7903 log.cc:826] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Log is configured to *not* fsync() on all Append() calls
I20250815 08:00:01.827886 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 08:00:01.828789 7903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Bootstrap complete.
I20250815 08:00:01.856529 7903 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } }
I20250815 08:00:01.857287 7903 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1319d17be64fcc96f00fdfcb0adb3a, State: Initialized, Role: FOLLOWER
I20250815 08:00:01.858045 7903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } }
I20250815 08:00:01.858628 7903 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250815 08:00:01.858913 7903 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250815 08:00:01.859267 7903 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 1 FOLLOWER]: Advancing to term 2
I20250815 08:00:01.864952 7903 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } }
I20250815 08:00:01.865723 7903 leader_election.cc:304] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f1319d17be64fcc96f00fdfcb0adb3a; no voters:
I20250815 08:00:01.867514 7903 leader_election.cc:290] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [CANDIDATE]: Term 2 election: Requested vote from peers
I20250815 08:00:01.867946 7907 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 2 FOLLOWER]: Leader election won for term 2
I20250815 08:00:01.871289 7907 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [term 2 LEADER]: Becoming Leader. State: Replica: 8f1319d17be64fcc96f00fdfcb0adb3a, State: Running, Role: LEADER
I20250815 08:00:01.872092 7907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } }
I20250815 08:00:01.872314 7903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: configured and running, proceeding with master startup.
I20250815 08:00:01.879210 7909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f1319d17be64fcc96f00fdfcb0adb3a. Latest consensus state: current_term: 2 leader_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } } }
I20250815 08:00:01.881737 7909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: This master's current role is: LEADER
I20250815 08:00:01.880971 7908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1319d17be64fcc96f00fdfcb0adb3a" member_type: VOTER last_known_addr { host: "127.1.121.254" port: 33557 } } }
I20250815 08:00:01.882102 7908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a [sys.catalog]: This master's current role is: LEADER
I20250815 08:00:01.884672 7914 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250815 08:00:01.895937 7914 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=6f0cd805fe2141f5985b597e9261cb3e]
I20250815 08:00:01.903744 7914 tablet_loader.cc:96] loaded metadata for tablet 5090f1bb5ffb42c09350308bbcef6940 (table pre_rebuild [id=6f0cd805fe2141f5985b597e9261cb3e])
I20250815 08:00:01.905045 7914 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250815 08:00:01.932749 7914 catalog_manager.cc:1349] Generated new cluster ID: e3ea089ef1da4814a3537ca653afd191
I20250815 08:00:01.933023 7914 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250815 08:00:01.995548 7914 catalog_manager.cc:1372] Generated new certificate authority record
I20250815 08:00:01.997416 7914 catalog_manager.cc:1506] Loading token signing keys...
I20250815 08:00:02.028224 7914 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Generated new TSK 0
I20250815 08:00:02.029210 7914 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250815 08:00:02.134480 7905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 08:00:02.134951 7905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 08:00:02.135440 7905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 08:00:02.165072 7905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 08:00:02.165894 7905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.193
I20250815 08:00:02.198239 7905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.193:35817
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.121.193
--webserver_port=32819
--tserver_master_addrs=127.1.121.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.193
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 08:00:02.199570 7905 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 08:00:02.201268 7905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 08:00:02.213001 7931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 08:00:03.321499 7899 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 7836
W20250815 08:00:03.322146 7899 debug-util.cc:398] Leaking SignalData structure 0x7b080006f7a0 after lost signal to thread 7902
W20250815 08:00:02.215060 7932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 08:00:03.615550 7930 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 7905
W20250815 08:00:03.816049 7905 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.602s user 0.487s sys 0.805s
W20250815 08:00:03.818912 7905 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.605s user 0.487s sys 0.805s
W20250815 08:00:03.819455 7934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 08:00:03.822436 7933 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1603 milliseconds
I20250815 08:00:03.822455 7905 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250815 08:00:03.823705 7905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 08:00:03.826036 7905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 08:00:03.827440 7905 hybrid_clock.cc:648] HybridClock initialized: now 1755244803827404 us; error 33 us; skew 500 ppm
I20250815 08:00:03.828202 7905 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 08:00:03.835250 7905 webserver.cc:480] Webserver started at http://127.1.121.193:32819/ using document root <none> and password file <none>
I20250815 08:00:03.836194 7905 fs_manager.cc:362] Metadata directory not provided
I20250815 08:00:03.836431 7905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 08:00:03.844192 7905 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250815 08:00:03.849015 7941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 08:00:03.850036 7905 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250815 08:00:03.850324 7905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
uuid: "283bb3462b894e23a649bc318b5011bb"
format_stamp: "Formatted at 2025-08-15 07:59:47 on dist-test-slave-dlgj"
I20250815 08:00:03.852195 7905 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 08:00:03.902882 7905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 08:00:03.904287 7905 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 08:00:03.904695 7905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 08:00:03.907541 7905 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 08:00:03.913982 7948 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 08:00:03.921015 7905 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 08:00:03.921291 7905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250815 08:00:03.921602 7905 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 08:00:03.925944 7905 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 08:00:03.926225 7905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250815 08:00:03.926529 7948 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Bootstrap starting.
I20250815 08:00:04.333398 7948 log.cc:826] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Log is configured to *not* fsync() on all Append() calls
I20250815 08:00:04.770401 8055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.193:35817 every 8 connection(s)
I20250815 08:00:04.771039 7905 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.193:35817
I20250815 08:00:04.774106 7905 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/info.pb
I20250815 08:00:04.779839 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 7905
I20250815 08:00:04.781847 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.194:34441
--local_ip_for_outbound_sockets=127.1.121.194
--tserver_master_addrs=127.1.121.254:33557
--webserver_port=35733
--webserver_interface=127.1.121.194
--builtin_ntp_servers=127.1.121.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 08:00:04.843410 8056 heartbeater.cc:344] Connected to a master server at 127.1.121.254:33557
I20250815 08:00:04.843907 8056 heartbeater.cc:461] Registering TS with master...
I20250815 08:00:04.845134 8056 heartbeater.cc:507] Master 127.1.121.254:33557 requested a full tablet report, sending...
I20250815 08:00:04.849308 7868 ts_manager.cc:194] Registered new tserver with Master: 283bb3462b894e23a649bc318b5011bb (127.1.121.193:35817)
I20250815 08:00:04.856910 7868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.193:58931
W20250815 08:00:05.228740 8060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 08:00:05.229393 8060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 08:00:05.230190 8060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 08:00:05.279949 8060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 08:00:05.281337 8060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.194
I20250815 08:00:05.333838 8060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.194:34441
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.121.194
--webserver_port=35733
--tserver_master_addrs=127.1.121.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.194
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 08:00:05.335462 8060 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 08:00:05.337335 8060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 08:00:05.350672 8066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 08:00:05.860862 8056 heartbeater.cc:499] Master 127.1.121.254:33557 was elected leader, sending a full tablet report...
W20250815 08:00:05.353022 8067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 08:00:06.642307 8060 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.291s user 0.407s sys 0.843s
W20250815 08:00:06.642611 8060 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.291s user 0.407s sys 0.843s
I20250815 08:00:06.652544 8060 server_base.cc:1047] running on GCE node
W20250815 08:00:06.654052 8071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 08:00:06.655117 8060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 08:00:06.660598 8060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 08:00:06.662024 8060 hybrid_clock.cc:648] HybridClock initialized: now 1755244806661949 us; error 78 us; skew 500 ppm
I20250815 08:00:06.662802 8060 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 08:00:06.672677 8060 webserver.cc:480] Webserver started at http://127.1.121.194:35733/ using document root <none> and password file <none>
I20250815 08:00:06.673558 8060 fs_manager.cc:362] Metadata directory not provided
I20250815 08:00:06.673811 8060 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 08:00:06.686955 8060 fs_manager.cc:714] Time spent opening directory manager: real 0.010s user 0.009s sys 0.001s
I20250815 08:00:06.692965 8076 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 08:00:06.693975 8060 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250815 08:00:06.694270 8060 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
uuid: "5f38cf428cd34649b8c24f8930910161"
format_stamp: "Formatted at 2025-08-15 07:59:49 on dist-test-slave-dlgj"
I20250815 08:00:06.696130 8060 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 08:00:06.789057 8060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 08:00:06.790432 8060 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 08:00:06.790838 8060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 08:00:06.793293 8060 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 08:00:06.799209 8083 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 08:00:06.809219 8060 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 08:00:06.809458 8060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.001s
I20250815 08:00:06.809700 8060 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 08:00:06.813866 8060 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 08:00:06.814074 8060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250815 08:00:06.814464 8083 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Bootstrap starting.
I20250815 08:00:06.986660 8060 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.194:34441
I20250815 08:00:06.986892 8189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.194:34441 every 8 connection(s)
I20250815 08:00:06.990705 8060 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/info.pb
I20250815 08:00:06.992797 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 8060
I20250815 08:00:06.997053 1511 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
/tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.121.195:42845
--local_ip_for_outbound_sockets=127.1.121.195
--tserver_master_addrs=127.1.121.254:33557
--webserver_port=38635
--webserver_interface=127.1.121.195
--builtin_ntp_servers=127.1.121.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250815 08:00:07.076930 8190 heartbeater.cc:344] Connected to a master server at 127.1.121.254:33557
I20250815 08:00:07.077378 8190 heartbeater.cc:461] Registering TS with master...
I20250815 08:00:07.078478 8190 heartbeater.cc:507] Master 127.1.121.254:33557 requested a full tablet report, sending...
I20250815 08:00:07.082319 7868 ts_manager.cc:194] Registered new tserver with Master: 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441)
I20250815 08:00:07.085263 7868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.194:58729
I20250815 08:00:07.110489 8083 log.cc:826] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Log is configured to *not* fsync() on all Append() calls
W20250815 08:00:07.472215 8194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250815 08:00:07.472766 8194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250815 08:00:07.473416 8194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250815 08:00:07.523728 8194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250815 08:00:07.524979 8194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.121.195
I20250815 08:00:07.580572 8194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.121.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.121.195:42845
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.121.195
--webserver_port=38635
--tserver_master_addrs=127.1.121.254:33557
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.121.195
--log_dir=/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 15 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7577
TSAN enabled
I20250815 08:00:07.582127 8194 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250815 08:00:07.584100 8194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250815 08:00:07.597800 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 08:00:07.802574 7948 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 08:00:07.803956 7948 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Bootstrap complete.
I20250815 08:00:07.806455 7948 ts_tablet_manager.cc:1397] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Time spent bootstrapping tablet: real 3.880s user 3.697s sys 0.084s
I20250815 08:00:07.829743 7948 raft_consensus.cc:357] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:07.834067 7948 raft_consensus.cc:738] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283bb3462b894e23a649bc318b5011bb, State: Initialized, Role: FOLLOWER
I20250815 08:00:07.835445 7948 consensus_queue.cc:260] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:07.853852 7948 ts_tablet_manager.cc:1428] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Time spent starting tablet: real 0.047s user 0.036s sys 0.012s
I20250815 08:00:08.089056 8190 heartbeater.cc:499] Master 127.1.121.254:33557 was elected leader, sending a full tablet report...
W20250815 08:00:07.598812 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250815 08:00:08.766142 8194 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.169s user 0.459s sys 0.685s
W20250815 08:00:08.766536 8194 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.170s user 0.459s sys 0.685s
I20250815 08:00:08.780445 8194 server_base.cc:1047] running on GCE node
W20250815 08:00:08.782006 8207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250815 08:00:08.783102 8194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250815 08:00:08.794027 8194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250815 08:00:08.795480 8194 hybrid_clock.cc:648] HybridClock initialized: now 1755244808795423 us; error 61 us; skew 500 ppm
I20250815 08:00:08.796211 8194 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250815 08:00:08.804816 8194 webserver.cc:480] Webserver started at http://127.1.121.195:38635/ using document root <none> and password file <none>
I20250815 08:00:08.805634 8194 fs_manager.cc:362] Metadata directory not provided
I20250815 08:00:08.805836 8194 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250815 08:00:08.813599 8194 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250815 08:00:08.818531 8212 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250815 08:00:08.819490 8194 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250815 08:00:08.819872 8194 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
uuid: "9abb03c51da8480ea4190a3d5fcbf1d0"
format_stamp: "Formatted at 2025-08-15 07:59:51 on dist-test-slave-dlgj"
I20250815 08:00:08.821782 8194 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250815 08:00:08.885002 8194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250815 08:00:08.886318 8194 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250815 08:00:08.886739 8194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250815 08:00:08.888949 8194 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250815 08:00:08.894744 8219 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250815 08:00:08.901062 8194 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250815 08:00:08.901261 8194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.001s
I20250815 08:00:08.901535 8194 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250815 08:00:08.905484 8194 ts_tablet_manager.cc:610] Registered 1 tablets
I20250815 08:00:08.905687 8194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20250815 08:00:08.905997 8219 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Bootstrap starting.
I20250815 08:00:09.090226 8194 rpc_server.cc:307] RPC server started. Bound to: 127.1.121.195:42845
I20250815 08:00:09.090515 8325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.121.195:42845 every 8 connection(s)
I20250815 08:00:09.093860 8194 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/info.pb
I20250815 08:00:09.094676 1511 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu as pid 8194
I20250815 08:00:09.139578 8326 heartbeater.cc:344] Connected to a master server at 127.1.121.254:33557
I20250815 08:00:09.140107 8326 heartbeater.cc:461] Registering TS with master...
I20250815 08:00:09.141395 8326 heartbeater.cc:507] Master 127.1.121.254:33557 requested a full tablet report, sending...
I20250815 08:00:09.145691 7868 ts_manager.cc:194] Registered new tserver with Master: 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845)
I20250815 08:00:09.148524 7868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.121.195:54867
I20250815 08:00:09.160979 1511 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250815 08:00:09.209623 8219 log.cc:826] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Log is configured to *not* fsync() on all Append() calls
I20250815 08:00:09.608551 8339 raft_consensus.cc:491] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 08:00:09.609074 8339 raft_consensus.cc:513] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:09.611814 8339 leader_election.cc:290] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845), 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441)
I20250815 08:00:09.642717 8281 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "283bb3462b894e23a649bc318b5011bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" is_pre_election: true
W20250815 08:00:09.653890 7944 leader_election.cc:343] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845): Illegal state: must be running to vote when last-logged opid is not known
I20250815 08:00:09.651816 8145 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "283bb3462b894e23a649bc318b5011bb" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "5f38cf428cd34649b8c24f8930910161" is_pre_election: true
W20250815 08:00:09.659785 7944 leader_election.cc:343] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441): Illegal state: must be running to vote when last-logged opid is not known
I20250815 08:00:09.660060 7944 leader_election.cc:304] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 283bb3462b894e23a649bc318b5011bb; no voters: 5f38cf428cd34649b8c24f8930910161, 9abb03c51da8480ea4190a3d5fcbf1d0
I20250815 08:00:09.660722 8339 raft_consensus.cc:2747] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250815 08:00:09.724074 8083 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 08:00:09.724721 8083 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Bootstrap complete.
I20250815 08:00:09.725857 8083 ts_tablet_manager.cc:1397] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Time spent bootstrapping tablet: real 2.912s user 2.784s sys 0.056s
I20250815 08:00:09.730279 8083 raft_consensus.cc:357] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:09.732308 8083 raft_consensus.cc:738] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f38cf428cd34649b8c24f8930910161, State: Initialized, Role: FOLLOWER
I20250815 08:00:09.732949 8083 consensus_queue.cc:260] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:09.736629 8083 ts_tablet_manager.cc:1428] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20250815 08:00:10.151964 8326 heartbeater.cc:499] Master 127.1.121.254:33557 was elected leader, sending a full tablet report...
I20250815 08:00:10.838475 8219 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250815 08:00:10.839121 8219 tablet_bootstrap.cc:492] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Bootstrap complete.
I20250815 08:00:10.840323 8219 ts_tablet_manager.cc:1397] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Time spent bootstrapping tablet: real 1.935s user 1.872s sys 0.044s
I20250815 08:00:10.844816 8219 raft_consensus.cc:357] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:10.846475 8219 raft_consensus.cc:738] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb03c51da8480ea4190a3d5fcbf1d0, State: Initialized, Role: FOLLOWER
I20250815 08:00:10.846999 8219 consensus_queue.cc:260] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:10.849092 8219 ts_tablet_manager.cc:1428] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20250815 08:00:11.357295 8347 raft_consensus.cc:491] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 08:00:11.357753 8347 raft_consensus.cc:513] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:11.359959 8347 leader_election.cc:290] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845), 283bb3462b894e23a649bc318b5011bb (127.1.121.193:35817)
I20250815 08:00:11.370630 8281 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "5f38cf428cd34649b8c24f8930910161" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" is_pre_election: true
I20250815 08:00:11.371171 8281 raft_consensus.cc:2466] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f38cf428cd34649b8c24f8930910161 in term 1.
I20250815 08:00:11.371765 8011 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "5f38cf428cd34649b8c24f8930910161" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "283bb3462b894e23a649bc318b5011bb" is_pre_election: true
I20250815 08:00:11.372340 8011 raft_consensus.cc:2466] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f38cf428cd34649b8c24f8930910161 in term 1.
I20250815 08:00:11.372339 8079 leader_election.cc:304] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f38cf428cd34649b8c24f8930910161, 9abb03c51da8480ea4190a3d5fcbf1d0; no voters:
I20250815 08:00:11.372967 8347 raft_consensus.cc:2802] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250815 08:00:11.373297 8347 raft_consensus.cc:491] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 08:00:11.373570 8347 raft_consensus.cc:3058] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Advancing to term 2
I20250815 08:00:11.379244 8347 raft_consensus.cc:513] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:11.380656 8347 leader_election.cc:290] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [CANDIDATE]: Term 2 election: Requested vote from peers 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845), 283bb3462b894e23a649bc318b5011bb (127.1.121.193:35817)
I20250815 08:00:11.381259 8281 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "5f38cf428cd34649b8c24f8930910161" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0"
I20250815 08:00:11.381531 8011 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5090f1bb5ffb42c09350308bbcef6940" candidate_uuid: "5f38cf428cd34649b8c24f8930910161" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "283bb3462b894e23a649bc318b5011bb"
I20250815 08:00:11.381697 8281 raft_consensus.cc:3058] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Advancing to term 2
I20250815 08:00:11.381946 8011 raft_consensus.cc:3058] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Advancing to term 2
I20250815 08:00:11.387023 8011 raft_consensus.cc:2466] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f38cf428cd34649b8c24f8930910161 in term 2.
I20250815 08:00:11.387027 8281 raft_consensus.cc:2466] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f38cf428cd34649b8c24f8930910161 in term 2.
I20250815 08:00:11.387796 8078 leader_election.cc:304] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283bb3462b894e23a649bc318b5011bb, 5f38cf428cd34649b8c24f8930910161; no voters:
I20250815 08:00:11.388381 8347 raft_consensus.cc:2802] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 2 FOLLOWER]: Leader election won for term 2
I20250815 08:00:11.389598 8347 raft_consensus.cc:695] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 2 LEADER]: Becoming Leader. State: Replica: 5f38cf428cd34649b8c24f8930910161, State: Running, Role: LEADER
I20250815 08:00:11.390347 8347 consensus_queue.cc:237] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:11.398617 7868 catalog_manager.cc:5582] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 reported cstate change: term changed from 0 to 2, leader changed from <none> to 5f38cf428cd34649b8c24f8930910161 (127.1.121.194), VOTER 283bb3462b894e23a649bc318b5011bb (127.1.121.193) added, VOTER 5f38cf428cd34649b8c24f8930910161 (127.1.121.194) added, VOTER 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195) added. New cstate: current_term: 2 leader_uuid: "5f38cf428cd34649b8c24f8930910161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } health_report { overall_health: HEALTHY } } }
I20250815 08:00:11.883740 8281 raft_consensus.cc:1273] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 2 FOLLOWER]: Refusing update from remote peer 5f38cf428cd34649b8c24f8930910161: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250815 08:00:11.884829 8347 consensus_queue.cc:1035] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250815 08:00:11.896231 8011 raft_consensus.cc:1273] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 2 FOLLOWER]: Refusing update from remote peer 5f38cf428cd34649b8c24f8930910161: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250815 08:00:11.897796 8363 consensus_queue.cc:1035] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250815 08:00:11.918021 8145 consensus_queue.cc:237] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:11.920806 8281 raft_consensus.cc:1273] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 2 FOLLOWER]: Refusing update from remote peer 5f38cf428cd34649b8c24f8930910161: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250815 08:00:11.921725 8363 consensus_queue.cc:1035] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250815 08:00:11.925410 8347 raft_consensus.cc:2953] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 283bb3462b894e23a649bc318b5011bb (127.1.121.193) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } } }
I20250815 08:00:11.926582 8281 raft_consensus.cc:2953] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 283bb3462b894e23a649bc318b5011bb (127.1.121.193) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } } }
I20250815 08:00:11.930971 7854 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5090f1bb5ffb42c09350308bbcef6940 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 08:00:11.933594 7868 catalog_manager.cc:5582] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 reported cstate change: config changed from index -1 to 207, VOTER 283bb3462b894e23a649bc318b5011bb (127.1.121.193) evicted. New cstate: current_term: 2 leader_uuid: "5f38cf428cd34649b8c24f8930910161" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } health_report { overall_health: HEALTHY } } }
I20250815 08:00:11.943821 8145 consensus_queue.cc:237] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:11.945858 8347 raft_consensus.cc:2953] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } } }
I20250815 08:00:11.952304 7854 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5090f1bb5ffb42c09350308bbcef6940 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250815 08:00:11.952361 7990 tablet_service.cc:1515] Processing DeleteTablet for tablet 5090f1bb5ffb42c09350308bbcef6940 with delete_type TABLET_DATA_TOMBSTONED (TS 283bb3462b894e23a649bc318b5011bb not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:43752
I20250815 08:00:11.954414 8373 tablet_replica.cc:331] stopping tablet replica
I20250815 08:00:11.954888 8373 raft_consensus.cc:2241] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 08:00:11.954646 7868 catalog_manager.cc:5582] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 reported cstate change: config changed from index 207 to 208, VOTER 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195) evicted. New cstate: current_term: 2 leader_uuid: "5f38cf428cd34649b8c24f8930910161" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } health_report { overall_health: HEALTHY } } }
I20250815 08:00:11.955256 8373 raft_consensus.cc:2270] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 08:00:11.969079 8261 tablet_service.cc:1515] Processing DeleteTablet for tablet 5090f1bb5ffb42c09350308bbcef6940 with delete_type TABLET_DATA_TOMBSTONED (TS 9abb03c51da8480ea4190a3d5fcbf1d0 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:35608
I20250815 08:00:11.971704 8375 tablet_replica.cc:331] stopping tablet replica
I20250815 08:00:11.972254 8375 raft_consensus.cc:2241] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250815 08:00:11.972668 8375 raft_consensus.cc:2270] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 08:00:11.972674 8373 ts_tablet_manager.cc:1905] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 08:00:11.981627 8373 ts_tablet_manager.cc:1918] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250815 08:00:11.981875 8373 log.cc:1199] T 5090f1bb5ffb42c09350308bbcef6940 P 283bb3462b894e23a649bc318b5011bb: Deleting WAL directory at /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/wal/wals/5090f1bb5ffb42c09350308bbcef6940
I20250815 08:00:11.983124 7853 catalog_manager.cc:4928] TS 283bb3462b894e23a649bc318b5011bb (127.1.121.193:35817): tablet 5090f1bb5ffb42c09350308bbcef6940 (table pre_rebuild [id=6f0cd805fe2141f5985b597e9261cb3e]) successfully deleted
I20250815 08:00:11.987697 8375 ts_tablet_manager.cc:1905] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250815 08:00:11.996946 8375 ts_tablet_manager.cc:1918] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250815 08:00:11.997229 8375 log.cc:1199] T 5090f1bb5ffb42c09350308bbcef6940 P 9abb03c51da8480ea4190a3d5fcbf1d0: Deleting WAL directory at /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/wal/wals/5090f1bb5ffb42c09350308bbcef6940
I20250815 08:00:11.998342 7854 catalog_manager.cc:4928] TS 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845): tablet 5090f1bb5ffb42c09350308bbcef6940 (table pre_rebuild [id=6f0cd805fe2141f5985b597e9261cb3e]) successfully deleted
W20250815 08:00:12.245790 1511 scanner-internal.cc:458] Time spent opening tablet: real 3.044s user 0.004s sys 0.004s
I20250815 08:00:12.776888 7990 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 08:00:12.786674 8125 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 08:00:12.798488 8261 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250815 08:00:12.816430 8186 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5a40 after lost signal to thread 8061
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8f1319d17be64fcc96f00fdfcb0adb3a | 127.1.121.254:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.1.121.212:36221 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
283bb3462b894e23a649bc318b5011bb | 127.1.121.193:35817 | HEALTHY | <none> | 0 | 0
5f38cf428cd34649b8c24f8930910161 | 127.1.121.194:34441 | HEALTHY | <none> | 1 | 0
9abb03c51da8480ea4190a3d5fcbf1d0 | 127.1.121.195:42845 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.121.193 | experimental | 127.1.121.193:35817
local_ip_for_outbound_sockets | 127.1.121.194 | experimental | 127.1.121.194:34441
local_ip_for_outbound_sockets | 127.1.121.195 | experimental | 127.1.121.195:42845
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/info.pb | hidden | 127.1.121.193:35817
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/info.pb | hidden | 127.1.121.194:34441
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/info.pb | hidden | 127.1.121.195:42845
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.1.121.212:36221 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 08:00:13.088222 1511 log_verifier.cc:126] Checking tablet 5090f1bb5ffb42c09350308bbcef6940
I20250815 08:00:13.420578 1511 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 5090f1bb5ffb42c09350308bbcef6940
I20250815 08:00:13.423632 7867 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51626:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250815 08:00:13.424191 7867 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51626:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250815 08:00:13.439723 7867 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 8f1319d17be64fcc96f00fdfcb0adb3a: Sending DeleteTablet for 1 replicas of tablet 5090f1bb5ffb42c09350308bbcef6940
I20250815 08:00:13.441825 1511 test_util.cc:276] Using random seed: -946251000
I20250815 08:00:13.441766 8125 tablet_service.cc:1515] Processing DeleteTablet for tablet 5090f1bb5ffb42c09350308bbcef6940 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-15 08:00:13 UTC) from {username='slave'} at 127.0.0.1:42480
I20250815 08:00:13.459450 8411 tablet_replica.cc:331] stopping tablet replica
I20250815 08:00:13.460276 8411 raft_consensus.cc:2241] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 2 LEADER]: Raft consensus shutting down.
I20250815 08:00:13.460935 8411 raft_consensus.cc:2270] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250815 08:00:13.496486 8411 ts_tablet_manager.cc:1905] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250815 08:00:13.511354 7867 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50688:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250815 08:00:13.514640 7867 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250815 08:00:13.517177 8411 ts_tablet_manager.cc:1918] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250815 08:00:13.517580 8411 log.cc:1199] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Deleting WAL directory at /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/wal/wals/5090f1bb5ffb42c09350308bbcef6940
I20250815 08:00:13.518512 8411 ts_tablet_manager.cc:1939] T 5090f1bb5ffb42c09350308bbcef6940 P 5f38cf428cd34649b8c24f8930910161: Deleting consensus metadata
I20250815 08:00:13.521538 7854 catalog_manager.cc:4928] TS 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441): tablet 5090f1bb5ffb42c09350308bbcef6940 (table pre_rebuild [id=6f0cd805fe2141f5985b597e9261cb3e]) successfully deleted
I20250815 08:00:13.539418 8261 tablet_service.cc:1468] Processing CreateTablet for tablet 3fa32ec44e324ae181019333f324803a (DEFAULT_TABLE table=post_rebuild [id=ba683b0c04a7406fb82d2d19f4152b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 08:00:13.540248 7990 tablet_service.cc:1468] Processing CreateTablet for tablet 3fa32ec44e324ae181019333f324803a (DEFAULT_TABLE table=post_rebuild [id=ba683b0c04a7406fb82d2d19f4152b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 08:00:13.540834 8261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa32ec44e324ae181019333f324803a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 08:00:13.541608 7990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa32ec44e324ae181019333f324803a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 08:00:13.543056 8125 tablet_service.cc:1468] Processing CreateTablet for tablet 3fa32ec44e324ae181019333f324803a (DEFAULT_TABLE table=post_rebuild [id=ba683b0c04a7406fb82d2d19f4152b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250815 08:00:13.544230 8125 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fa32ec44e324ae181019333f324803a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250815 08:00:13.564254 8418 tablet_bootstrap.cc:492] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0: Bootstrap starting.
I20250815 08:00:13.564625 8419 tablet_bootstrap.cc:492] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb: Bootstrap starting.
I20250815 08:00:13.580526 8418 tablet_bootstrap.cc:654] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0: Neither blocks nor log segments found. Creating new log.
I20250815 08:00:13.581102 8420 tablet_bootstrap.cc:492] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161: Bootstrap starting.
I20250815 08:00:13.582538 8419 tablet_bootstrap.cc:654] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb: Neither blocks nor log segments found. Creating new log.
I20250815 08:00:13.586444 8420 tablet_bootstrap.cc:654] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161: Neither blocks nor log segments found. Creating new log.
I20250815 08:00:13.600785 8418 tablet_bootstrap.cc:492] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0: No bootstrap required, opened a new log
I20250815 08:00:13.601243 8418 ts_tablet_manager.cc:1397] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0: Time spent bootstrapping tablet: real 0.040s user 0.020s sys 0.005s
I20250815 08:00:13.601919 8420 tablet_bootstrap.cc:492] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161: No bootstrap required, opened a new log
I20250815 08:00:13.602344 8420 ts_tablet_manager.cc:1397] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.010s
I20250815 08:00:13.603677 8418 raft_consensus.cc:357] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.604581 8418 raft_consensus.cc:383] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 08:00:13.604887 8418 raft_consensus.cc:738] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abb03c51da8480ea4190a3d5fcbf1d0, State: Initialized, Role: FOLLOWER
I20250815 08:00:13.604671 8420 raft_consensus.cc:357] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.605355 8420 raft_consensus.cc:383] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 08:00:13.605645 8420 raft_consensus.cc:738] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f38cf428cd34649b8c24f8930910161, State: Initialized, Role: FOLLOWER
I20250815 08:00:13.605604 8418 consensus_queue.cc:260] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.606375 8420 consensus_queue.cc:260] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.611169 8419 tablet_bootstrap.cc:492] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb: No bootstrap required, opened a new log
I20250815 08:00:13.611649 8419 ts_tablet_manager.cc:1397] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.011s
I20250815 08:00:13.614192 8419 raft_consensus.cc:357] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.614948 8419 raft_consensus.cc:383] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250815 08:00:13.615227 8419 raft_consensus.cc:738] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283bb3462b894e23a649bc318b5011bb, State: Initialized, Role: FOLLOWER
I20250815 08:00:13.616858 8419 consensus_queue.cc:260] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.625880 8419 ts_tablet_manager.cc:1428] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20250815 08:00:13.633260 8418 ts_tablet_manager.cc:1428] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0: Time spent starting tablet: real 0.032s user 0.004s sys 0.004s
I20250815 08:00:13.638999 8420 ts_tablet_manager.cc:1428] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161: Time spent starting tablet: real 0.036s user 0.010s sys 0.012s
I20250815 08:00:13.755156 8425 raft_consensus.cc:491] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 08:00:13.755647 8425 raft_consensus.cc:513] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.764626 8425 leader_election.cc:290] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283bb3462b894e23a649bc318b5011bb (127.1.121.193:35817), 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441)
I20250815 08:00:13.795389 8426 raft_consensus.cc:491] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250815 08:00:13.795814 8426 raft_consensus.cc:513] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.797644 8426 leader_election.cc:290] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845), 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441)
I20250815 08:00:13.799279 8281 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa32ec44e324ae181019333f324803a" candidate_uuid: "283bb3462b894e23a649bc318b5011bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" is_pre_election: true
I20250815 08:00:13.799798 8281 raft_consensus.cc:2466] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283bb3462b894e23a649bc318b5011bb in term 0.
I20250815 08:00:13.802188 7944 leader_election.cc:304] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283bb3462b894e23a649bc318b5011bb, 9abb03c51da8480ea4190a3d5fcbf1d0; no voters:
I20250815 08:00:13.802919 8426 raft_consensus.cc:2802] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250815 08:00:13.803172 8426 raft_consensus.cc:491] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250815 08:00:13.803426 8426 raft_consensus.cc:3058] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 0 FOLLOWER]: Advancing to term 1
W20250815 08:00:13.804316 8191 tablet.cc:2378] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 08:00:13.808777 8145 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa32ec44e324ae181019333f324803a" candidate_uuid: "283bb3462b894e23a649bc318b5011bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f38cf428cd34649b8c24f8930910161" is_pre_election: true
I20250815 08:00:13.809324 8145 raft_consensus.cc:2466] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283bb3462b894e23a649bc318b5011bb in term 0.
W20250815 08:00:13.812134 8059 tablet.cc:2378] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 08:00:13.812248 8426 raft_consensus.cc:513] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.815289 8281 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa32ec44e324ae181019333f324803a" candidate_uuid: "283bb3462b894e23a649bc318b5011bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0"
I20250815 08:00:13.815773 8281 raft_consensus.cc:3058] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 0 FOLLOWER]: Advancing to term 1
I20250815 08:00:13.821072 8011 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa32ec44e324ae181019333f324803a" candidate_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283bb3462b894e23a649bc318b5011bb" is_pre_election: true
I20250815 08:00:13.821909 8011 raft_consensus.cc:2391] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9abb03c51da8480ea4190a3d5fcbf1d0 in current term 1: Already voted for candidate 283bb3462b894e23a649bc318b5011bb in this term.
I20250815 08:00:13.824628 8281 raft_consensus.cc:2466] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283bb3462b894e23a649bc318b5011bb in term 1.
I20250815 08:00:13.825273 8145 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa32ec44e324ae181019333f324803a" candidate_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f38cf428cd34649b8c24f8930910161" is_pre_election: true
I20250815 08:00:13.825757 8145 raft_consensus.cc:2466] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9abb03c51da8480ea4190a3d5fcbf1d0 in term 0.
I20250815 08:00:13.826972 8215 leader_election.cc:304] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f38cf428cd34649b8c24f8930910161, 9abb03c51da8480ea4190a3d5fcbf1d0; no voters:
I20250815 08:00:13.827950 8425 raft_consensus.cc:2762] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250815 08:00:13.829123 7944 leader_election.cc:304] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283bb3462b894e23a649bc318b5011bb, 9abb03c51da8480ea4190a3d5fcbf1d0; no voters:
I20250815 08:00:13.834975 8426 leader_election.cc:290] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [CANDIDATE]: Term 1 election: Requested vote from peers 9abb03c51da8480ea4190a3d5fcbf1d0 (127.1.121.195:42845), 5f38cf428cd34649b8c24f8930910161 (127.1.121.194:34441)
I20250815 08:00:13.836217 8426 raft_consensus.cc:2802] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 1 FOLLOWER]: Leader election won for term 1
I20250815 08:00:13.837157 8426 raft_consensus.cc:695] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [term 1 LEADER]: Becoming Leader. State: Replica: 283bb3462b894e23a649bc318b5011bb, State: Running, Role: LEADER
I20250815 08:00:13.838497 8426 consensus_queue.cc:237] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } }
I20250815 08:00:13.843106 8145 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fa32ec44e324ae181019333f324803a" candidate_uuid: "283bb3462b894e23a649bc318b5011bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f38cf428cd34649b8c24f8930910161"
I20250815 08:00:13.843570 8145 raft_consensus.cc:3058] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [term 0 FOLLOWER]: Advancing to term 1
I20250815 08:00:13.850366 8145 raft_consensus.cc:2466] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283bb3462b894e23a649bc318b5011bb in term 1.
I20250815 08:00:13.866869 7867 catalog_manager.cc:5582] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 283bb3462b894e23a649bc318b5011bb (127.1.121.193). New cstate: current_term: 1 leader_uuid: "283bb3462b894e23a649bc318b5011bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283bb3462b894e23a649bc318b5011bb" member_type: VOTER last_known_addr { host: "127.1.121.193" port: 35817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 } health_report { overall_health: UNKNOWN } } }
W20250815 08:00:13.881000 8329 tablet.cc:2378] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250815 08:00:14.160701 8281 raft_consensus.cc:1273] T 3fa32ec44e324ae181019333f324803a P 9abb03c51da8480ea4190a3d5fcbf1d0 [term 1 FOLLOWER]: Refusing update from remote peer 283bb3462b894e23a649bc318b5011bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 08:00:14.162488 8145 raft_consensus.cc:1273] T 3fa32ec44e324ae181019333f324803a P 5f38cf428cd34649b8c24f8930910161 [term 1 FOLLOWER]: Refusing update from remote peer 283bb3462b894e23a649bc318b5011bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250815 08:00:14.164472 8426 consensus_queue.cc:1035] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f38cf428cd34649b8c24f8930910161" member_type: VOTER last_known_addr { host: "127.1.121.194" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 08:00:14.165052 8431 consensus_queue.cc:1035] T 3fa32ec44e324ae181019333f324803a P 283bb3462b894e23a649bc318b5011bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9abb03c51da8480ea4190a3d5fcbf1d0" member_type: VOTER last_known_addr { host: "127.1.121.195" port: 42845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250815 08:00:14.201970 8439 mvcc.cc:204] Tried to move back new op lower bound from 7189482758783078400 to 7189482757542834176. Current Snapshot: MvccSnapshot[applied={T|T < 7189482758783078400}]
I20250815 08:00:14.206316 8441 mvcc.cc:204] Tried to move back new op lower bound from 7189482758783078400 to 7189482757542834176. Current Snapshot: MvccSnapshot[applied={T|T < 7189482758783078400}]
I20250815 08:00:18.946082 7990 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250815 08:00:18.952780 8125 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250815 08:00:18.955778 8261 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8f1319d17be64fcc96f00fdfcb0adb3a | 127.1.121.254:33557 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.1.121.212:36221 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
283bb3462b894e23a649bc318b5011bb | 127.1.121.193:35817 | HEALTHY | <none> | 1 | 0
5f38cf428cd34649b8c24f8930910161 | 127.1.121.194:34441 | HEALTHY | <none> | 0 | 0
9abb03c51da8480ea4190a3d5fcbf1d0 | 127.1.121.195:42845 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.121.193 | experimental | 127.1.121.193:35817
local_ip_for_outbound_sockets | 127.1.121.194 | experimental | 127.1.121.194:34441
local_ip_for_outbound_sockets | 127.1.121.195 | experimental | 127.1.121.195:42845
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-0/data/info.pb | hidden | 127.1.121.193:35817
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-1/data/info.pb | hidden | 127.1.121.194:34441
server_dump_info_path | /tmp/dist-test-taskfKDZCv/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755244639110353-1511-0/minicluster-data/ts-2/data/info.pb | hidden | 127.1.121.195:42845
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.1.121.212:36221 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250815 08:00:19.168725 1511 log_verifier.cc:126] Checking tablet 3fa32ec44e324ae181019333f324803a
I20250815 08:00:19.867921 1511 log_verifier.cc:177] Verified matching terms for 206 ops in tablet 3fa32ec44e324ae181019333f324803a
I20250815 08:00:19.869004 1511 log_verifier.cc:126] Checking tablet 5090f1bb5ffb42c09350308bbcef6940
I20250815 08:00:19.869371 1511 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 5090f1bb5ffb42c09350308bbcef6940
I20250815 08:00:19.892871 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 7905
I20250815 08:00:19.931730 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 8060
I20250815 08:00:19.969828 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 8194
I20250815 08:00:20.001734 1511 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskfKDZCv/build/tsan/bin/kudu with pid 7835
2025-08-15T08:00:20Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35738 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35738 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (180889 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST