Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250628 20:03:26.968483 5521 test_util.cc:276] Using random seed: 1236010638
I20250628 20:03:26.975726 5521 ts_itest-base.cc:115] Starting cluster with:
I20250628 20:03:26.975797 5521 ts_itest-base.cc:116] --------------
I20250628 20:03:26.975816 5521 ts_itest-base.cc:117] 4 tablet servers
I20250628 20:03:26.975834 5521 ts_itest-base.cc:118] 3 replicas per TS
I20250628 20:03:26.975855 5521 ts_itest-base.cc:119] --------------
2025-06-28T20:03:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T20:03:26Z Disabled control of system clock
I20250628 20:03:26.984992 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:32805
--webserver_interface=127.5.100.126
--webserver_port=0
--builtin_ntp_servers=127.5.100.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:32805 with env {}
W20250628 20:03:27.086452 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:27.086674 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:27.086714 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:27.089793 5533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:27.089848 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:27.089867 5533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:27.089883 5533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:27.093521 5533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:32805
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:32805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:27.094609 5533 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:27.095700 5533 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:27.102350 5538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.102421 5539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:27.102496 5533 server_base.cc:1048] running on GCE node
W20250628 20:03:27.102429 5541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:27.103080 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:27.104023 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:27.105191 5533 hybrid_clock.cc:648] HybridClock initialized: now 1751141007105168 us; error 27 us; skew 500 ppm
I20250628 20:03:27.107183 5533 webserver.cc:469] Webserver started at http://127.5.100.126:33231/ using document root <none> and password file <none>
I20250628 20:03:27.107748 5533 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:27.107805 5533 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:27.107995 5533 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:27.109639 5533 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4fe350d3dc264fd99d181e1ede1b0b30"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.110169 5533 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4fe350d3dc264fd99d181e1ede1b0b30"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.113440 5533 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:27.115613 5547 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.116614 5533 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250628 20:03:27.116750 5533 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4fe350d3dc264fd99d181e1ede1b0b30"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.116845 5533 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:27.133062 5533 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:27.133800 5533 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:27.133987 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:27.140961 5533 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:32805
I20250628 20:03:27.140971 5599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:32805 every 8 connection(s)
I20250628 20:03:27.142058 5533 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:27.144641 5600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:27.149876 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Bootstrap starting.
I20250628 20:03:27.150305 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 5533
I20250628 20:03:27.150449 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 20:03:27.152303 5600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:27.153077 5600 log.cc:826] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:27.155131 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: No bootstrap required, opened a new log
I20250628 20:03:27.158001 5600 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } }
I20250628 20:03:27.158252 5600 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:27.158313 5600 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe350d3dc264fd99d181e1ede1b0b30, State: Initialized, Role: FOLLOWER
I20250628 20:03:27.158810 5600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } }
I20250628 20:03:27.158974 5600 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:27.159042 5600 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:27.159148 5600 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:27.160108 5600 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } }
I20250628 20:03:27.160468 5600 leader_election.cc:304] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe350d3dc264fd99d181e1ede1b0b30; no voters:
I20250628 20:03:27.160784 5600 leader_election.cc:290] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:27.160949 5605 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:27.161134 5605 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe350d3dc264fd99d181e1ede1b0b30, State: Running, Role: LEADER
I20250628 20:03:27.161489 5605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } }
I20250628 20:03:27.162038 5600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:27.162736 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe350d3dc264fd99d181e1ede1b0b30. Latest consensus state: current_term: 1 leader_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } } }
I20250628 20:03:27.162882 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:27.163254 5613 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:27.163940 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } } }
I20250628 20:03:27.164099 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:27.166378 5613 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:27.172466 5613 catalog_manager.cc:1349] Generated new cluster ID: bbe009a723ff4eb29834e34a2f4ac8ab
I20250628 20:03:27.172549 5613 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:27.195052 5613 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:27.196009 5613 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:27.209410 5613 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Generated new TSK 0
I20250628 20:03:27.210247 5613 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:27.215782 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:0
--local_ip_for_outbound_sockets=127.5.100.65
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32805
--builtin_ntp_servers=127.5.100.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 20:03:27.319546 5624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:27.319768 5624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:27.319813 5624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:27.323114 5624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:27.323235 5624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:27.326987 5624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32805
--never_fsync=true
--heap_profile_path=/tmp/kudu.5624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:27.328073 5624 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:27.329202 5624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:27.336589 5630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.336611 5629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.336609 5632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:27.336828 5624 server_base.cc:1048] running on GCE node
I20250628 20:03:27.337330 5624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:27.337965 5624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:27.339121 5624 hybrid_clock.cc:648] HybridClock initialized: now 1751141007339099 us; error 36 us; skew 500 ppm
I20250628 20:03:27.340965 5624 webserver.cc:469] Webserver started at http://127.5.100.65:40623/ using document root <none> and password file <none>
I20250628 20:03:27.341533 5624 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:27.341591 5624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:27.341811 5624 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:27.343456 5624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "18c8d6fa347a42be910384376ebbc1d8"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.343940 5624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "18c8d6fa347a42be910384376ebbc1d8"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.347362 5624 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:27.349545 5638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.350594 5624 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:27.350708 5624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "18c8d6fa347a42be910384376ebbc1d8"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.350795 5624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:27.369247 5624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:27.370028 5624 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:27.370203 5624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:27.370769 5624 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:27.371722 5624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:27.371770 5624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.371821 5624 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:27.371850 5624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.380635 5624 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:36773
I20250628 20:03:27.380673 5751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:36773 every 8 connection(s)
I20250628 20:03:27.381603 5624 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 20:03:27.390489 5752 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32805
I20250628 20:03:27.390753 5752 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:27.391346 5752 heartbeater.cc:507] Master 127.5.100.126:32805 requested a full tablet report, sending...
I20250628 20:03:27.391570 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 5624
I20250628 20:03:27.391667 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 20:03:27.392828 5564 ts_manager.cc:194] Registered new tserver with Master: 18c8d6fa347a42be910384376ebbc1d8 (127.5.100.65:36773)
I20250628 20:03:27.393781 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:0
--local_ip_for_outbound_sockets=127.5.100.66
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32805
--builtin_ntp_servers=127.5.100.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:27.394141 5564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:48871
W20250628 20:03:27.496354 5755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:27.496580 5755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:27.496623 5755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:27.500114 5755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:27.500236 5755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:27.504014 5755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32805
--never_fsync=true
--heap_profile_path=/tmp/kudu.5755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:27.505122 5755 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:27.506289 5755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:27.513732 5763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.513727 5760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.513958 5761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:27.514037 5755 server_base.cc:1048] running on GCE node
I20250628 20:03:27.514417 5755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:27.514982 5755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:27.516129 5755 hybrid_clock.cc:648] HybridClock initialized: now 1751141007516117 us; error 25 us; skew 500 ppm
I20250628 20:03:27.518052 5755 webserver.cc:469] Webserver started at http://127.5.100.66:37459/ using document root <none> and password file <none>
I20250628 20:03:27.518623 5755 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:27.518683 5755 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:27.518862 5755 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:27.520478 5755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5bb443de4a394df08d3b4a1dcb2e532c"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.520954 5755 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5bb443de4a394df08d3b4a1dcb2e532c"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.524443 5755 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250628 20:03:27.526652 5769 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.527714 5755 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:27.527827 5755 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5bb443de4a394df08d3b4a1dcb2e532c"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.527917 5755 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:27.551359 5755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:27.552052 5755 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:27.552242 5755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:27.552824 5755 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:27.553866 5755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:27.553920 5755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.553974 5755 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:27.554001 5755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.562706 5755 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:41127
I20250628 20:03:27.562736 5882 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:41127 every 8 connection(s)
I20250628 20:03:27.563657 5755 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 20:03:27.569939 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 5755
I20250628 20:03:27.570037 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 20:03:27.572152 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:0
--local_ip_for_outbound_sockets=127.5.100.67
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32805
--builtin_ntp_servers=127.5.100.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:27.572510 5883 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32805
I20250628 20:03:27.572739 5883 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:27.573297 5883 heartbeater.cc:507] Master 127.5.100.126:32805 requested a full tablet report, sending...
I20250628 20:03:27.574316 5564 ts_manager.cc:194] Registered new tserver with Master: 5bb443de4a394df08d3b4a1dcb2e532c (127.5.100.66:41127)
I20250628 20:03:27.574823 5564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:43871
W20250628 20:03:27.677747 5886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:27.677989 5886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:27.678036 5886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:27.681278 5886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:27.681407 5886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:27.685258 5886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32805
--never_fsync=true
--heap_profile_path=/tmp/kudu.5886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:27.686419 5886 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:27.687538 5886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:27.695405 5891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.695423 5894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.695382 5892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:27.695724 5886 server_base.cc:1048] running on GCE node
I20250628 20:03:27.696120 5886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:27.696719 5886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:27.697871 5886 hybrid_clock.cc:648] HybridClock initialized: now 1751141007697848 us; error 36 us; skew 500 ppm
I20250628 20:03:27.699831 5886 webserver.cc:469] Webserver started at http://127.5.100.67:38633/ using document root <none> and password file <none>
I20250628 20:03:27.700417 5886 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:27.700477 5886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:27.700654 5886 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:27.702325 5886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "acc1ea86678b45ec9f59c5dea80c9af5"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.702795 5886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "acc1ea86678b45ec9f59c5dea80c9af5"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.706171 5886 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:27.708392 5900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.709442 5886 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:27.709543 5886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "acc1ea86678b45ec9f59c5dea80c9af5"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.709635 5886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:27.722282 5886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:27.722954 5886 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:27.723121 5886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:27.723683 5886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:27.724624 5886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:27.724670 5886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.724715 5886 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:27.724742 5886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.733315 5886 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:41457
I20250628 20:03:27.733338 6013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:41457 every 8 connection(s)
I20250628 20:03:27.734339 5886 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 20:03:27.738799 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 5886
I20250628 20:03:27.738910 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 20:03:27.741066 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.68:0
--local_ip_for_outbound_sockets=127.5.100.68
--webserver_interface=127.5.100.68
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32805
--builtin_ntp_servers=127.5.100.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:27.743806 6014 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32805
I20250628 20:03:27.744097 6014 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:27.744697 6014 heartbeater.cc:507] Master 127.5.100.126:32805 requested a full tablet report, sending...
I20250628 20:03:27.745785 5564 ts_manager.cc:194] Registered new tserver with Master: acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67:41457)
I20250628 20:03:27.746353 5564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:48367
W20250628 20:03:27.849248 6017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:27.849478 6017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:27.849527 6017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:27.852727 6017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:27.852888 6017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.68
I20250628 20:03:27.856884 6017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.100.68
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32805
--never_fsync=true
--heap_profile_path=/tmp/kudu.6017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.68
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:27.858067 6017 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:27.859273 6017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:27.866848 6023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.866894 6022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:27.866848 6025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:27.867113 6017 server_base.cc:1048] running on GCE node
I20250628 20:03:27.867544 6017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:27.868117 6017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:27.869266 6017 hybrid_clock.cc:648] HybridClock initialized: now 1751141007869248 us; error 30 us; skew 500 ppm
I20250628 20:03:27.871145 6017 webserver.cc:469] Webserver started at http://127.5.100.68:40809/ using document root <none> and password file <none>
I20250628 20:03:27.871696 6017 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:27.871753 6017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:27.871937 6017 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:27.873484 6017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5ca4c1d0dd174385b725f5fd58e5930c"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.874022 6017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5ca4c1d0dd174385b725f5fd58e5930c"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.877228 6017 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:27.879364 6031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.880311 6017 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250628 20:03:27.880416 6017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5ca4c1d0dd174385b725f5fd58e5930c"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:27.880502 6017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:27.896807 6017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:27.897476 6017 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:27.897639 6017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:27.898262 6017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:27.899235 6017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:27.899283 6017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.899330 6017 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:27.899358 6017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:27.907790 6017 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.68:41523
I20250628 20:03:27.907805 6144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.68:41523 every 8 connection(s)
I20250628 20:03:27.908788 6017 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250628 20:03:27.909709 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 6017
I20250628 20:03:27.909819 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250628 20:03:27.917649 6145 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32805
I20250628 20:03:27.917907 6145 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:27.918411 6145 heartbeater.cc:507] Master 127.5.100.126:32805 requested a full tablet report, sending...
I20250628 20:03:27.919310 5564 ts_manager.cc:194] Registered new tserver with Master: 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68:41523)
I20250628 20:03:27.919842 5564 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.68:59925
I20250628 20:03:27.924155 5521 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250628 20:03:27.935451 5564 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33500:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 20:03:27.952503 5948 tablet_service.cc:1468] Processing CreateTablet for tablet 763057be0f1b464081ab9c24bff2f285 (DEFAULT_TABLE table=TestTable [id=b483367d59a94a2eb109a32f1c25f4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:27.952503 5817 tablet_service.cc:1468] Processing CreateTablet for tablet 763057be0f1b464081ab9c24bff2f285 (DEFAULT_TABLE table=TestTable [id=b483367d59a94a2eb109a32f1c25f4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:27.952503 5686 tablet_service.cc:1468] Processing CreateTablet for tablet 763057be0f1b464081ab9c24bff2f285 (DEFAULT_TABLE table=TestTable [id=b483367d59a94a2eb109a32f1c25f4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:27.953469 5948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763057be0f1b464081ab9c24bff2f285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:27.953480 5817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763057be0f1b464081ab9c24bff2f285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:27.953480 5686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763057be0f1b464081ab9c24bff2f285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:27.958555 6162 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: Bootstrap starting.
I20250628 20:03:27.959757 6163 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8: Bootstrap starting.
I20250628 20:03:27.960911 6162 tablet_bootstrap.cc:654] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:27.961493 6163 tablet_bootstrap.cc:654] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:27.961756 6162 log.cc:826] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:27.962035 6164 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c: Bootstrap starting.
I20250628 20:03:27.962297 6163 log.cc:826] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:27.963353 6162 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: No bootstrap required, opened a new log
I20250628 20:03:27.963550 6162 ts_tablet_manager.cc:1397] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250628 20:03:27.964136 6163 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8: No bootstrap required, opened a new log
I20250628 20:03:27.964140 6164 tablet_bootstrap.cc:654] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:27.964324 6163 ts_tablet_manager.cc:1397] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250628 20:03:27.964897 6164 log.cc:826] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:27.966099 6162 raft_consensus.cc:357] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:27.966287 6162 raft_consensus.cc:383] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:27.966365 6162 raft_consensus.cc:738] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc1ea86678b45ec9f59c5dea80c9af5, State: Initialized, Role: FOLLOWER
I20250628 20:03:27.966786 6164 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c: No bootstrap required, opened a new log
I20250628 20:03:27.966828 6162 consensus_queue.cc:260] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:27.967039 6164 ts_tablet_manager.cc:1397] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250628 20:03:27.967198 6163 raft_consensus.cc:357] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:27.967406 6163 raft_consensus.cc:383] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:27.967496 6163 raft_consensus.cc:738] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c8d6fa347a42be910384376ebbc1d8, State: Initialized, Role: FOLLOWER
I20250628 20:03:27.967803 6162 ts_tablet_manager.cc:1428] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250628 20:03:27.967906 6163 consensus_queue.cc:260] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:27.968158 6014 heartbeater.cc:499] Master 127.5.100.126:32805 was elected leader, sending a full tablet report...
I20250628 20:03:27.968645 5752 heartbeater.cc:499] Master 127.5.100.126:32805 was elected leader, sending a full tablet report...
I20250628 20:03:27.968806 6163 ts_tablet_manager.cc:1428] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20250628 20:03:27.969944 6164 raft_consensus.cc:357] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:27.970108 6164 raft_consensus.cc:383] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:27.970183 6164 raft_consensus.cc:738] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb443de4a394df08d3b4a1dcb2e532c, State: Initialized, Role: FOLLOWER
I20250628 20:03:27.970515 6164 consensus_queue.cc:260] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:27.971109 5883 heartbeater.cc:499] Master 127.5.100.126:32805 was elected leader, sending a full tablet report...
I20250628 20:03:27.971215 6164 ts_tablet_manager.cc:1428] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250628 20:03:27.985224 6015 tablet.cc:2378] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 20:03:28.064774 5884 tablet.cc:2378] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:28.100771 6170 raft_consensus.cc:491] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:28.101039 6170 raft_consensus.cc:513] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:28.102277 6170 leader_election.cc:290] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c8d6fa347a42be910384376ebbc1d8 (127.5.100.65:36773), acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67:41457)
I20250628 20:03:28.106571 5968 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "763057be0f1b464081ab9c24bff2f285" candidate_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" is_pre_election: true
I20250628 20:03:28.106683 5706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "763057be0f1b464081ab9c24bff2f285" candidate_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c8d6fa347a42be910384376ebbc1d8" is_pre_election: true
I20250628 20:03:28.106860 5968 raft_consensus.cc:2466] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb443de4a394df08d3b4a1dcb2e532c in term 0.
I20250628 20:03:28.106966 5706 raft_consensus.cc:2466] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bb443de4a394df08d3b4a1dcb2e532c in term 0.
I20250628 20:03:28.107328 5771 leader_election.cc:304] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bb443de4a394df08d3b4a1dcb2e532c, acc1ea86678b45ec9f59c5dea80c9af5; no voters:
I20250628 20:03:28.107592 6170 raft_consensus.cc:2802] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 20:03:28.107676 6170 raft_consensus.cc:491] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 20:03:28.107728 6170 raft_consensus.cc:3058] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:28.108709 6170 raft_consensus.cc:513] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:28.109082 6170 leader_election.cc:290] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [CANDIDATE]: Term 1 election: Requested vote from peers 18c8d6fa347a42be910384376ebbc1d8 (127.5.100.65:36773), acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67:41457)
I20250628 20:03:28.109459 5706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "763057be0f1b464081ab9c24bff2f285" candidate_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c8d6fa347a42be910384376ebbc1d8"
I20250628 20:03:28.109459 5968 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "763057be0f1b464081ab9c24bff2f285" candidate_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1ea86678b45ec9f59c5dea80c9af5"
I20250628 20:03:28.109596 5706 raft_consensus.cc:3058] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:28.109596 5968 raft_consensus.cc:3058] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:28.110749 5968 raft_consensus.cc:2466] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bb443de4a394df08d3b4a1dcb2e532c in term 1.
I20250628 20:03:28.110757 5706 raft_consensus.cc:2466] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bb443de4a394df08d3b4a1dcb2e532c in term 1.
I20250628 20:03:28.111090 5772 leader_election.cc:304] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c8d6fa347a42be910384376ebbc1d8, 5bb443de4a394df08d3b4a1dcb2e532c; no voters:
I20250628 20:03:28.111263 6170 raft_consensus.cc:2802] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:28.111474 6170 raft_consensus.cc:695] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [term 1 LEADER]: Becoming Leader. State: Replica: 5bb443de4a394df08d3b4a1dcb2e532c, State: Running, Role: LEADER
I20250628 20:03:28.111826 6170 consensus_queue.cc:237] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:28.114889 5558 catalog_manager.cc:5582] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bb443de4a394df08d3b4a1dcb2e532c (127.5.100.66). New cstate: current_term: 1 leader_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } health_report { overall_health: HEALTHY } } }
W20250628 20:03:28.132838 5753 tablet.cc:2378] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:28.136286 5521 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250628 20:03:28.138049 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 18c8d6fa347a42be910384376ebbc1d8 to finish bootstrapping
I20250628 20:03:28.143760 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5bb443de4a394df08d3b4a1dcb2e532c to finish bootstrapping
I20250628 20:03:28.148739 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver acc1ea86678b45ec9f59c5dea80c9af5 to finish bootstrapping
I20250628 20:03:28.154042 5521 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250628 20:03:28.155514 5521 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "763057be0f1b464081ab9c24bff2f285"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8"
rpc_addresses {
host: "127.5.100.65"
port: 36773
}
}
ts_infos {
permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5"
rpc_addresses {
host: "127.5.100.67"
port: 41457
}
}
ts_infos {
permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c"
rpc_addresses {
host: "127.5.100.66"
port: 41127
}
}
I20250628 20:03:28.700523 6177 consensus_queue.cc:1035] T 763057be0f1b464081ab9c24bff2f285 P 5bb443de4a394df08d3b4a1dcb2e532c [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:28.706175 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 5755
I20250628 20:03:28.712111 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 5533
I20250628 20:03:28.718250 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:32805
--webserver_interface=127.5.100.126
--webserver_port=33231
--builtin_ntp_servers=127.5.100.84:36213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:32805 with env {}
W20250628 20:03:28.820469 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:28.820695 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:28.820749 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:28.823905 6184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:28.823980 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:28.824007 6184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:28.824029 6184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:28.827814 6184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:32805
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:32805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=33231
--never_fsync=true
--heap_profile_path=/tmp/kudu.6184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:28.828922 6184 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:28.830082 6184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:28.837081 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:28.837172 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:28.837083 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:28.837337 6184 server_base.cc:1048] running on GCE node
I20250628 20:03:28.837908 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:28.838807 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:28.839957 6184 hybrid_clock.cc:648] HybridClock initialized: now 1751141008839944 us; error 28 us; skew 500 ppm
I20250628 20:03:28.841881 6184 webserver.cc:469] Webserver started at http://127.5.100.126:33231/ using document root <none> and password file <none>
I20250628 20:03:28.842449 6184 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:28.842523 6184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:28.846203 6184 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:28.848950 6198 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:28.850304 6184 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:28.850432 6184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4fe350d3dc264fd99d181e1ede1b0b30"
format_stamp: "Formatted at 2025-06-28 20:03:27 on dist-test-slave-rmcj"
I20250628 20:03:28.850931 6184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:28.865346 6184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:28.866075 6184 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:28.866232 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:28.873219 6184 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:32805
I20250628 20:03:28.873238 6250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:32805 every 8 connection(s)
I20250628 20:03:28.874436 6184 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:28.878271 6251 sys_catalog.cc:263] Verifying existing consensus state
I20250628 20:03:28.880182 6251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Bootstrap starting.
I20250628 20:03:28.883500 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 6184
I20250628 20:03:28.883642 5521 kudu-admin-test.cc:735] Forcing unsafe config change on tserver acc1ea86678b45ec9f59c5dea80c9af5
I20250628 20:03:28.894258 6251 log.cc:826] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:28.901808 6251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:28.902139 6251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Bootstrap complete.
I20250628 20:03:28.905339 6251 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } }
I20250628 20:03:28.905674 6251 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe350d3dc264fd99d181e1ede1b0b30, State: Initialized, Role: FOLLOWER
I20250628 20:03:28.906184 6251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } }
I20250628 20:03:28.906294 6251 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:28.906353 6251 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:28.906453 6251 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:28.907969 6251 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } }
I20250628 20:03:28.908308 6251 leader_election.cc:304] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fe350d3dc264fd99d181e1ede1b0b30; no voters:
I20250628 20:03:28.908591 6251 leader_election.cc:290] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 20:03:28.908757 6255 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 20:03:28.909027 6255 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [term 2 LEADER]: Becoming Leader. State: Replica: 4fe350d3dc264fd99d181e1ede1b0b30, State: Running, Role: LEADER
I20250628 20:03:28.909377 6255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } }
I20250628 20:03:28.909827 6251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:28.910549 6255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } } }
I20250628 20:03:28.910756 6255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:28.911092 6255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fe350d3dc264fd99d181e1ede1b0b30. Latest consensus state: current_term: 2 leader_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fe350d3dc264fd99d181e1ede1b0b30" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32805 } } }
I20250628 20:03:28.911235 6255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:28.911540 6264 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:28.915256 6264 catalog_manager.cc:671] Loaded metadata for table TestTable [id=b483367d59a94a2eb109a32f1c25f4d2]
I20250628 20:03:28.916580 6264 tablet_loader.cc:96] loaded metadata for tablet 763057be0f1b464081ab9c24bff2f285 (table TestTable [id=b483367d59a94a2eb109a32f1c25f4d2])
I20250628 20:03:28.916749 6264 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:28.917366 6264 catalog_manager.cc:1261] Loaded cluster ID: bbe009a723ff4eb29834e34a2f4ac8ab
I20250628 20:03:28.917423 6264 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:28.919198 6264 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:28.919803 6264 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4fe350d3dc264fd99d181e1ede1b0b30: Loaded TSK: 0
I20250628 20:03:28.920742 6264 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:28.921862 6145 heartbeater.cc:499] Master 127.5.100.126:32805 was elected leader, sending a full tablet report...
I20250628 20:03:28.930055 6215 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" instance_seqno: 1751141007905429) as {username='slave'} at 127.5.100.68:60831; Asking this server to re-register.
I20250628 20:03:28.931070 6145 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:28.931272 6145 heartbeater.cc:507] Master 127.5.100.126:32805 requested a full tablet report, sending...
I20250628 20:03:28.932435 6215 ts_manager.cc:194] Registered new tserver with Master: 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68:41523)
I20250628 20:03:28.978457 6215 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" instance_seqno: 1751141007378206) as {username='slave'} at 127.5.100.65:42295; Asking this server to re-register.
I20250628 20:03:28.979023 5752 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:28.979235 5752 heartbeater.cc:507] Master 127.5.100.126:32805 requested a full tablet report, sending...
I20250628 20:03:28.980301 6215 ts_manager.cc:194] Registered new tserver with Master: 18c8d6fa347a42be910384376ebbc1d8 (127.5.100.65:36773)
W20250628 20:03:29.009821 6253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:29.010071 6253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:29.013365 6253 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250628 20:03:29.037596 5968 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "acc1ea86678b45ec9f59c5dea80c9af5"
tablet_id: "763057be0f1b464081ab9c24bff2f285"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8"
}
peers {
permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5"
}
}
from {username='slave'} at 127.0.0.1:60034
W20250628 20:03:29.038103 5968 raft_consensus.cc:2216] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } unsafe_config_change: true
I20250628 20:03:29.038343 5968 raft_consensus.cc:3058] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:29.044008 6215 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" instance_seqno: 1751141007730895) as {username='slave'} at 127.5.100.67:34577; Asking this server to re-register.
I20250628 20:03:29.044476 6014 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:29.044652 6014 heartbeater.cc:507] Master 127.5.100.126:32805 requested a full tablet report, sending...
I20250628 20:03:29.045416 6215 ts_manager.cc:194] Registered new tserver with Master: acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67:41457)
I20250628 20:03:30.096352 6294 raft_consensus.cc:491] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:30.096483 6294 raft_consensus.cc:513] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } }
I20250628 20:03:30.097476 6294 leader_election.cc:290] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67:41457), 5bb443de4a394df08d3b4a1dcb2e532c (127.5.100.66:41127)
W20250628 20:03:30.098227 5641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.100.66:41127: connect: Connection refused (error 111)
W20250628 20:03:30.099064 5641 leader_election.cc:336] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5bb443de4a394df08d3b4a1dcb2e532c (127.5.100.66:41127): Network error: Client connection negotiation failed: client connection to 127.5.100.66:41127: connect: Connection refused (error 111)
I20250628 20:03:30.100944 5968 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "763057be0f1b464081ab9c24bff2f285" candidate_uuid: "18c8d6fa347a42be910384376ebbc1d8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" is_pre_election: true
I20250628 20:03:30.101393 5640 leader_election.cc:304] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c8d6fa347a42be910384376ebbc1d8; no voters: 5bb443de4a394df08d3b4a1dcb2e532c, acc1ea86678b45ec9f59c5dea80c9af5
I20250628 20:03:30.101617 6294 raft_consensus.cc:2747] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250628 20:03:30.540920 6299 raft_consensus.cc:491] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250628 20:03:30.541052 6299 raft_consensus.cc:513] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } unsafe_config_change: true
I20250628 20:03:30.542016 6299 leader_election.cc:290] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 18c8d6fa347a42be910384376ebbc1d8 (127.5.100.65:36773)
I20250628 20:03:30.545312 5706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "763057be0f1b464081ab9c24bff2f285" candidate_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "18c8d6fa347a42be910384376ebbc1d8" is_pre_election: true
I20250628 20:03:30.545457 5706 raft_consensus.cc:2466] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc1ea86678b45ec9f59c5dea80c9af5 in term 1.
I20250628 20:03:30.545814 5903 leader_election.cc:304] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18c8d6fa347a42be910384376ebbc1d8, acc1ea86678b45ec9f59c5dea80c9af5; no voters:
I20250628 20:03:30.546051 6299 raft_consensus.cc:2802] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250628 20:03:30.546113 6299 raft_consensus.cc:491] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250628 20:03:30.546157 6299 raft_consensus.cc:3058] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 2 FOLLOWER]: Advancing to term 3
I20250628 20:03:30.546984 6299 raft_consensus.cc:513] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } unsafe_config_change: true
I20250628 20:03:30.547269 6299 leader_election.cc:290] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [CANDIDATE]: Term 3 election: Requested vote from peers 18c8d6fa347a42be910384376ebbc1d8 (127.5.100.65:36773)
I20250628 20:03:30.547641 5706 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "763057be0f1b464081ab9c24bff2f285" candidate_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "18c8d6fa347a42be910384376ebbc1d8"
I20250628 20:03:30.547747 5706 raft_consensus.cc:3058] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 1 FOLLOWER]: Advancing to term 3
I20250628 20:03:30.548465 5706 raft_consensus.cc:2466] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc1ea86678b45ec9f59c5dea80c9af5 in term 3.
I20250628 20:03:30.548717 5903 leader_election.cc:304] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 18c8d6fa347a42be910384376ebbc1d8, acc1ea86678b45ec9f59c5dea80c9af5; no voters:
I20250628 20:03:30.548878 6299 raft_consensus.cc:2802] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 3 FOLLOWER]: Leader election won for term 3
I20250628 20:03:30.549043 6299 raft_consensus.cc:695] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 3 LEADER]: Becoming Leader. State: Replica: acc1ea86678b45ec9f59c5dea80c9af5, State: Running, Role: LEADER
I20250628 20:03:30.549311 6299 consensus_queue.cc:237] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } unsafe_config_change: true
I20250628 20:03:30.551460 6215 catalog_manager.cc:5582] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 reported cstate change: term changed from 1 to 3, leader changed from 5bb443de4a394df08d3b4a1dcb2e532c (127.5.100.66) to acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67), now has a pending config: VOTER 18c8d6fa347a42be910384376ebbc1d8 (127.5.100.65), VOTER acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67). New cstate: current_term: 3 leader_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bb443de4a394df08d3b4a1dcb2e532c" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 41127 } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } unsafe_config_change: true }
I20250628 20:03:30.976744 5706 raft_consensus.cc:1273] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 3 FOLLOWER]: Refusing update from remote peer acc1ea86678b45ec9f59c5dea80c9af5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250628 20:03:30.977334 6299 consensus_queue.cc:1035] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:30.981432 6302 raft_consensus.cc:2953] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 5bb443de4a394df08d3b4a1dcb2e532c (127.5.100.66) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } unsafe_config_change: true }
I20250628 20:03:30.981851 5706 raft_consensus.cc:2953] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 3 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 5bb443de4a394df08d3b4a1dcb2e532c (127.5.100.66) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } unsafe_config_change: true }
I20250628 20:03:30.983927 6215 catalog_manager.cc:5582] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 reported cstate change: config changed from index -1 to 2, VOTER 5bb443de4a394df08d3b4a1dcb2e532c (127.5.100.66) evicted, no longer has a pending config: VOTER 18c8d6fa347a42be910384376ebbc1d8 (127.5.100.65), VOTER acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67). New cstate: current_term: 3 leader_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250628 20:03:30.986470 6215 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 763057be0f1b464081ab9c24bff2f285 on TS 5bb443de4a394df08d3b4a1dcb2e532c: Not found: failed to reset TS proxy: Could not find TS for UUID 5bb443de4a394df08d3b4a1dcb2e532c
I20250628 20:03:30.992542 5968 consensus_queue.cc:237] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: NON_VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: true } } unsafe_config_change: true
I20250628 20:03:30.994262 5706 raft_consensus.cc:1273] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 3 FOLLOWER]: Refusing update from remote peer acc1ea86678b45ec9f59c5dea80c9af5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250628 20:03:30.994699 6299 consensus_queue.cc:1035] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250628 20:03:30.996470 6299 raft_consensus.cc:2953] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: NON_VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: true } } unsafe_config_change: true }
I20250628 20:03:30.996856 5706 raft_consensus.cc:2953] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: NON_VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: true } } unsafe_config_change: true }
W20250628 20:03:30.998126 5901 consensus_peers.cc:489] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 -> Peer 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68:41523): Couldn't send request to peer 5ca4c1d0dd174385b725f5fd58e5930c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 763057be0f1b464081ab9c24bff2f285. This is attempt 1: this message will repeat every 5th retry.
I20250628 20:03:30.998773 6200 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 763057be0f1b464081ab9c24bff2f285 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250628 20:03:30.999186 6214 catalog_manager.cc:5582] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 reported cstate change: config changed from index 2 to 4, NON_VOTER 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68) added. New cstate: current_term: 3 leader_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: NON_VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: true } } unsafe_config_change: true }
W20250628 20:03:31.039294 6200 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 763057be0f1b464081ab9c24bff2f285 on TS 5bb443de4a394df08d3b4a1dcb2e532c failed: Not found: failed to reset TS proxy: Could not find TS for UUID 5bb443de4a394df08d3b4a1dcb2e532c
I20250628 20:03:31.527226 6317 ts_tablet_manager.cc:927] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: Initiating tablet copy from peer acc1ea86678b45ec9f59c5dea80c9af5 (127.5.100.67:41457)
I20250628 20:03:31.527722 6317 tablet_copy_client.cc:323] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.100.67:41457
I20250628 20:03:31.531199 5988 tablet_copy_service.cc:140] P acc1ea86678b45ec9f59c5dea80c9af5: Received BeginTabletCopySession request for tablet 763057be0f1b464081ab9c24bff2f285 from peer 5ca4c1d0dd174385b725f5fd58e5930c ({username='slave'} at 127.5.100.68:53053)
I20250628 20:03:31.531324 5988 tablet_copy_service.cc:161] P acc1ea86678b45ec9f59c5dea80c9af5: Beginning new tablet copy session on tablet 763057be0f1b464081ab9c24bff2f285 from peer 5ca4c1d0dd174385b725f5fd58e5930c at {username='slave'} at 127.5.100.68:53053: session id = 5ca4c1d0dd174385b725f5fd58e5930c-763057be0f1b464081ab9c24bff2f285
I20250628 20:03:31.532421 5988 tablet_copy_source_session.cc:215] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 20:03:31.535367 6317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 763057be0f1b464081ab9c24bff2f285. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:31.540000 6317 tablet_copy_client.cc:806] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: tablet copy: Starting download of 0 data blocks...
I20250628 20:03:31.540194 6317 tablet_copy_client.cc:670] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: tablet copy: Starting download of 1 WAL segments...
I20250628 20:03:31.541839 6317 tablet_copy_client.cc:538] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 20:03:31.543421 6317 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: Bootstrap starting.
I20250628 20:03:31.547809 6317 log.cc:826] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:31.550459 6317 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:31.550704 6317 tablet_bootstrap.cc:492] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: Bootstrap complete.
I20250628 20:03:31.550925 6317 ts_tablet_manager.cc:1397] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20250628 20:03:31.553503 6317 raft_consensus.cc:357] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: NON_VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: true } } unsafe_config_change: true
I20250628 20:03:31.553735 6317 raft_consensus.cc:738] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5ca4c1d0dd174385b725f5fd58e5930c, State: Initialized, Role: LEARNER
I20250628 20:03:31.554026 6317 consensus_queue.cc:260] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: NON_VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: true } } unsafe_config_change: true
I20250628 20:03:31.554827 6317 ts_tablet_manager.cc:1428] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250628 20:03:31.555773 5988 tablet_copy_service.cc:342] P acc1ea86678b45ec9f59c5dea80c9af5: Request end of tablet copy session 5ca4c1d0dd174385b725f5fd58e5930c-763057be0f1b464081ab9c24bff2f285 received from {username='slave'} at 127.5.100.68:53053
I20250628 20:03:31.555889 5988 tablet_copy_service.cc:434] P acc1ea86678b45ec9f59c5dea80c9af5: ending tablet copy session 5ca4c1d0dd174385b725f5fd58e5930c-763057be0f1b464081ab9c24bff2f285 on tablet 763057be0f1b464081ab9c24bff2f285 with peer 5ca4c1d0dd174385b725f5fd58e5930c
I20250628 20:03:32.093372 6099 raft_consensus.cc:1215] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[]
W20250628 20:03:32.205062 6200 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 763057be0f1b464081ab9c24bff2f285 on TS 5bb443de4a394df08d3b4a1dcb2e532c failed: Not found: failed to reset TS proxy: Could not find TS for UUID 5bb443de4a394df08d3b4a1dcb2e532c
I20250628 20:03:32.676294 6326 raft_consensus.cc:1062] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5: attempting to promote NON_VOTER 5ca4c1d0dd174385b725f5fd58e5930c to VOTER
I20250628 20:03:32.676743 6326 consensus_queue.cc:237] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: false } } unsafe_config_change: true
I20250628 20:03:32.678081 5706 raft_consensus.cc:1273] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 3 FOLLOWER]: Refusing update from remote peer acc1ea86678b45ec9f59c5dea80c9af5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250628 20:03:32.678138 6099 raft_consensus.cc:1273] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c [term 3 LEARNER]: Refusing update from remote peer acc1ea86678b45ec9f59c5dea80c9af5: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250628 20:03:32.678592 6326 consensus_queue.cc:1035] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250628 20:03:32.678802 6328 consensus_queue.cc:1035] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250628 20:03:32.680534 6326 raft_consensus.cc:2953] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: false } } unsafe_config_change: true }
I20250628 20:03:32.680999 5706 raft_consensus.cc:2953] T 763057be0f1b464081ab9c24bff2f285 P 18c8d6fa347a42be910384376ebbc1d8 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: false } } unsafe_config_change: true }
I20250628 20:03:32.682088 6099 raft_consensus.cc:2953] T 763057be0f1b464081ab9c24bff2f285 P 5ca4c1d0dd174385b725f5fd58e5930c [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: false } } unsafe_config_change: true }
I20250628 20:03:32.682931 6215 catalog_manager.cc:5582] T 763057be0f1b464081ab9c24bff2f285 P acc1ea86678b45ec9f59c5dea80c9af5 reported cstate change: config changed from index 4 to 5, 5ca4c1d0dd174385b725f5fd58e5930c (127.5.100.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 41457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 41523 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250628 20:03:32.808315 5521 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250628 20:03:32.819721 5521 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "763057be0f1b464081ab9c24bff2f285"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "18c8d6fa347a42be910384376ebbc1d8"
rpc_addresses {
host: "127.5.100.65"
port: 36773
}
}
ts_infos {
permanent_uuid: "acc1ea86678b45ec9f59c5dea80c9af5"
rpc_addresses {
host: "127.5.100.67"
port: 41457
}
}
ts_infos {
permanent_uuid: "5ca4c1d0dd174385b725f5fd58e5930c"
rpc_addresses {
host: "127.5.100.68"
port: 41523
}
}
I20250628 20:03:32.820830 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 5624
I20250628 20:03:32.826467 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 5886
I20250628 20:03:32.832077 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 6017
I20250628 20:03:32.837208 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 6184
2025-06-28T20:03:32Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (5891 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250628 20:03:32.859148 5521 test_util.cc:276] Using random seed: 1241901312
I20250628 20:03:32.859911 5521 ts_itest-base.cc:115] Starting cluster with:
I20250628 20:03:32.859952 5521 ts_itest-base.cc:116] --------------
I20250628 20:03:32.859968 5521 ts_itest-base.cc:117] 3 tablet servers
I20250628 20:03:32.859982 5521 ts_itest-base.cc:118] 3 replicas per TS
I20250628 20:03:32.859994 5521 ts_itest-base.cc:119] --------------
2025-06-28T20:03:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T20:03:32Z Disabled control of system clock
I20250628 20:03:32.867753 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:43367
--webserver_interface=127.5.100.126
--webserver_port=0
--builtin_ntp_servers=127.5.100.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:43367
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250628 20:03:32.972137 6346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:32.972386 6346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:32.972445 6346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:32.975690 6346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:32.975770 6346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:32.975796 6346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:32.975818 6346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:32.979712 6346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:43367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:32.980868 6346 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:32.982051 6346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:32.988695 6354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:32.988729 6352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:32.988695 6351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:32.988799 6346 server_base.cc:1048] running on GCE node
I20250628 20:03:32.989452 6346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:32.990442 6346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:32.991600 6346 hybrid_clock.cc:648] HybridClock initialized: now 1751141012991584 us; error 28 us; skew 500 ppm
I20250628 20:03:32.993593 6346 webserver.cc:469] Webserver started at http://127.5.100.126:41759/ using document root <none> and password file <none>
I20250628 20:03:32.994200 6346 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:32.994257 6346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:32.994442 6346 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:32.996124 6346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd"
format_stamp: "Formatted at 2025-06-28 20:03:32 on dist-test-slave-rmcj"
I20250628 20:03:32.996620 6346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd"
format_stamp: "Formatted at 2025-06-28 20:03:32 on dist-test-slave-rmcj"
I20250628 20:03:33.000078 6346 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:33.002341 6360 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.003428 6346 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:33.003562 6346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd"
format_stamp: "Formatted at 2025-06-28 20:03:32 on dist-test-slave-rmcj"
I20250628 20:03:33.003660 6346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:33.025633 6346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:33.026412 6346 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:33.026592 6346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:33.033862 6412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:43367 every 8 connection(s)
I20250628 20:03:33.033862 6346 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:43367
I20250628 20:03:33.034982 6346 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:33.037616 6413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:33.042975 6413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd: Bootstrap starting.
I20250628 20:03:33.043009 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 6346
I20250628 20:03:33.043129 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 20:03:33.045459 6413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:33.046263 6413 log.cc:826] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:33.048101 6413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd: No bootstrap required, opened a new log
I20250628 20:03:33.050988 6413 raft_consensus.cc:357] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 43367 } }
I20250628 20:03:33.051183 6413 raft_consensus.cc:383] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:33.051218 6413 raft_consensus.cc:738] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78b48d3d5a74b0ebf7c431cc1e263cd, State: Initialized, Role: FOLLOWER
I20250628 20:03:33.051653 6413 consensus_queue.cc:260] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 43367 } }
I20250628 20:03:33.051784 6413 raft_consensus.cc:397] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:33.051844 6413 raft_consensus.cc:491] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:33.051962 6413 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:33.052712 6413 raft_consensus.cc:513] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 43367 } }
I20250628 20:03:33.053035 6413 leader_election.cc:304] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d78b48d3d5a74b0ebf7c431cc1e263cd; no voters:
I20250628 20:03:33.053313 6413 leader_election.cc:290] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:33.053472 6418 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:33.053728 6418 raft_consensus.cc:695] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [term 1 LEADER]: Becoming Leader. State: Replica: d78b48d3d5a74b0ebf7c431cc1e263cd, State: Running, Role: LEADER
I20250628 20:03:33.054234 6418 consensus_queue.cc:237] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 43367 } }
I20250628 20:03:33.054394 6413 sys_catalog.cc:564] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:33.055873 6420 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d78b48d3d5a74b0ebf7c431cc1e263cd. Latest consensus state: current_term: 1 leader_uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 43367 } } }
I20250628 20:03:33.056095 6420 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:33.055966 6419 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78b48d3d5a74b0ebf7c431cc1e263cd" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 43367 } } }
I20250628 20:03:33.056223 6419 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd [sys.catalog]: This master's current role is: LEADER
W20250628 20:03:33.059098 6433 catalog_manager.cc:1560] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250628 20:03:33.059170 6433 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250628 20:03:33.059243 6432 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:33.059952 6432 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:33.065531 6432 catalog_manager.cc:1349] Generated new cluster ID: 8642716ed13840d0a39cd3cc93de3f71
I20250628 20:03:33.065605 6432 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:33.077659 6432 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:33.078624 6432 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:33.083904 6432 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d78b48d3d5a74b0ebf7c431cc1e263cd: Generated new TSK 0
I20250628 20:03:33.084617 6432 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:33.089129 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:0
--local_ip_for_outbound_sockets=127.5.100.65
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:43367
--builtin_ntp_servers=127.5.100.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250628 20:03:33.195586 6437 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 20:03:33.195832 6437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:33.195881 6437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:33.195945 6437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:33.199298 6437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:33.199434 6437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:33.203261 6437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:43367
--never_fsync=true
--heap_profile_path=/tmp/kudu.6437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:33.204362 6437 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:33.205509 6437 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:33.213629 6445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:33.213653 6443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:33.213656 6442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:33.214211 6437 server_base.cc:1048] running on GCE node
I20250628 20:03:33.214599 6437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:33.215174 6437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:33.216321 6437 hybrid_clock.cc:648] HybridClock initialized: now 1751141013216305 us; error 25 us; skew 500 ppm
I20250628 20:03:33.218339 6437 webserver.cc:469] Webserver started at http://127.5.100.65:45685/ using document root <none> and password file <none>
I20250628 20:03:33.218900 6437 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:33.218958 6437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:33.219146 6437 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:33.220698 6437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9353763a94f34212a8e08d0726c4518e"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.221153 6437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9353763a94f34212a8e08d0726c4518e"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.224450 6437 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:33.226696 6451 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.227700 6437 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:33.227818 6437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9353763a94f34212a8e08d0726c4518e"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.227973 6437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:33.265781 6437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:33.266502 6437 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:33.266686 6437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:33.267280 6437 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:33.268270 6437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:33.268319 6437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.268374 6437 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:33.268402 6437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.277113 6437 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:44931
I20250628 20:03:33.277130 6564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:44931 every 8 connection(s)
I20250628 20:03:33.278149 6437 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 20:03:33.285267 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 6437
I20250628 20:03:33.285387 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 20:03:33.287304 6565 heartbeater.cc:344] Connected to a master server at 127.5.100.126:43367
I20250628 20:03:33.287552 6565 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:33.287586 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:0
--local_ip_for_outbound_sockets=127.5.100.66
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:43367
--builtin_ntp_servers=127.5.100.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250628 20:03:33.288139 6565 heartbeater.cc:507] Master 127.5.100.126:43367 requested a full tablet report, sending...
I20250628 20:03:33.289535 6377 ts_manager.cc:194] Registered new tserver with Master: 9353763a94f34212a8e08d0726c4518e (127.5.100.65:44931)
I20250628 20:03:33.290725 6377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:60471
W20250628 20:03:33.394181 6568 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 20:03:33.394423 6568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:33.394444 6568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:33.394493 6568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:33.397727 6568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:33.397842 6568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:33.401603 6568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:43367
--never_fsync=true
--heap_profile_path=/tmp/kudu.6568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:33.402740 6568 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:33.403880 6568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:33.411659 6576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:33.411650 6574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:33.411816 6568 server_base.cc:1048] running on GCE node
W20250628 20:03:33.411664 6573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:33.412261 6568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:33.412794 6568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:33.413933 6568 hybrid_clock.cc:648] HybridClock initialized: now 1751141013413913 us; error 35 us; skew 500 ppm
I20250628 20:03:33.415621 6568 webserver.cc:469] Webserver started at http://127.5.100.66:43083/ using document root <none> and password file <none>
I20250628 20:03:33.416169 6568 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:33.416224 6568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:33.416407 6568 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:33.418011 6568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b4461325368e4b8dabdd3baaadf17248"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.418462 6568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b4461325368e4b8dabdd3baaadf17248"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.421723 6568 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:33.423772 6582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.424764 6568 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:33.424873 6568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b4461325368e4b8dabdd3baaadf17248"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.424968 6568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:33.440896 6568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:33.441603 6568 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:33.441810 6568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:33.442394 6568 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:33.443378 6568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:33.443425 6568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.443480 6568 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:33.443508 6568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.452790 6568 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:33013
I20250628 20:03:33.452822 6695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:33013 every 8 connection(s)
I20250628 20:03:33.453815 6568 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 20:03:33.454819 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 6568
I20250628 20:03:33.454929 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 20:03:33.457298 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:0
--local_ip_for_outbound_sockets=127.5.100.67
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:43367
--builtin_ntp_servers=127.5.100.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250628 20:03:33.464334 6696 heartbeater.cc:344] Connected to a master server at 127.5.100.126:43367
I20250628 20:03:33.464651 6696 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:33.465407 6696 heartbeater.cc:507] Master 127.5.100.126:43367 requested a full tablet report, sending...
I20250628 20:03:33.466536 6377 ts_manager.cc:194] Registered new tserver with Master: b4461325368e4b8dabdd3baaadf17248 (127.5.100.66:33013)
I20250628 20:03:33.467119 6377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:41029
W20250628 20:03:33.563758 6699 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 20:03:33.564020 6699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:33.564043 6699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:33.564106 6699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:33.567400 6699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:33.567529 6699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:33.571295 6699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:43367
--never_fsync=true
--heap_profile_path=/tmp/kudu.6699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:33.572415 6699 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:33.573550 6699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:33.581136 6704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:33.581210 6699 server_base.cc:1048] running on GCE node
W20250628 20:03:33.581120 6705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:33.581123 6707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:33.581751 6699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:33.582319 6699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:33.583484 6699 hybrid_clock.cc:648] HybridClock initialized: now 1751141013583466 us; error 30 us; skew 500 ppm
I20250628 20:03:33.585383 6699 webserver.cc:469] Webserver started at http://127.5.100.67:42435/ using document root <none> and password file <none>
I20250628 20:03:33.585980 6699 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:33.586038 6699 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:33.586218 6699 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:33.587786 6699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0107549ec7014c85819739c61e35466e"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.588258 6699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0107549ec7014c85819739c61e35466e"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.591518 6699 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:33.593590 6713 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.594596 6699 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:33.594702 6699 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0107549ec7014c85819739c61e35466e"
format_stamp: "Formatted at 2025-06-28 20:03:33 on dist-test-slave-rmcj"
I20250628 20:03:33.594787 6699 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:33.620824 6699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:33.621529 6699 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:33.621732 6699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:33.622354 6699 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:33.623340 6699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:33.623389 6699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.623442 6699 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:33.623471 6699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:33.632159 6699 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:34983
I20250628 20:03:33.632206 6826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:34983 every 8 connection(s)
I20250628 20:03:33.633143 6699 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 20:03:33.641996 6827 heartbeater.cc:344] Connected to a master server at 127.5.100.126:43367
I20250628 20:03:33.642274 6827 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:33.642848 6827 heartbeater.cc:507] Master 127.5.100.126:43367 requested a full tablet report, sending...
I20250628 20:03:33.643250 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 6699
I20250628 20:03:33.643342 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 20:03:33.643961 6377 ts_manager.cc:194] Registered new tserver with Master: 0107549ec7014c85819739c61e35466e (127.5.100.67:34983)
I20250628 20:03:33.644517 6377 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:42055
I20250628 20:03:33.645269 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:33.653234 6376 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56386:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 20:03:33.655663 6376 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 20:03:33.671290 6761 tablet_service.cc:1468] Processing CreateTablet for tablet 5a813da1389c4358b4f85a204062cbe5 (DEFAULT_TABLE table=TestTable [id=f814ceffe84b4f50b79ba000d82ab95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:33.671376 6630 tablet_service.cc:1468] Processing CreateTablet for tablet 5a813da1389c4358b4f85a204062cbe5 (DEFAULT_TABLE table=TestTable [id=f814ceffe84b4f50b79ba000d82ab95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:33.671376 6499 tablet_service.cc:1468] Processing CreateTablet for tablet 5a813da1389c4358b4f85a204062cbe5 (DEFAULT_TABLE table=TestTable [id=f814ceffe84b4f50b79ba000d82ab95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:33.672318 6761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a813da1389c4358b4f85a204062cbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:33.672320 6499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a813da1389c4358b4f85a204062cbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:33.672320 6630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a813da1389c4358b4f85a204062cbe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:33.677330 6844 tablet_bootstrap.cc:492] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e: Bootstrap starting.
I20250628 20:03:33.678686 6846 tablet_bootstrap.cc:492] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248: Bootstrap starting.
I20250628 20:03:33.678686 6845 tablet_bootstrap.cc:492] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e: Bootstrap starting.
I20250628 20:03:33.679474 6844 tablet_bootstrap.cc:654] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:33.680251 6844 log.cc:826] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:33.680792 6845 tablet_bootstrap.cc:654] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:33.680792 6846 tablet_bootstrap.cc:654] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:33.681591 6845 log.cc:826] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:33.681592 6846 log.cc:826] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:33.682020 6844 tablet_bootstrap.cc:492] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e: No bootstrap required, opened a new log
I20250628 20:03:33.682235 6844 ts_tablet_manager.cc:1397] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250628 20:03:33.683302 6845 tablet_bootstrap.cc:492] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e: No bootstrap required, opened a new log
I20250628 20:03:33.683480 6846 tablet_bootstrap.cc:492] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248: No bootstrap required, opened a new log
I20250628 20:03:33.683483 6845 ts_tablet_manager.cc:1397] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250628 20:03:33.683645 6846 ts_tablet_manager.cc:1397] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250628 20:03:33.684675 6844 raft_consensus.cc:357] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:33.684863 6844 raft_consensus.cc:738] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0107549ec7014c85819739c61e35466e, State: Initialized, Role: FOLLOWER
I20250628 20:03:33.685179 6844 consensus_queue.cc:260] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:33.685820 6827 heartbeater.cc:499] Master 127.5.100.126:43367 was elected leader, sending a full tablet report...
I20250628 20:03:33.686431 6846 raft_consensus.cc:357] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:33.686431 6845 raft_consensus.cc:357] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:33.686636 6846 raft_consensus.cc:738] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4461325368e4b8dabdd3baaadf17248, State: Initialized, Role: FOLLOWER
I20250628 20:03:33.686643 6845 raft_consensus.cc:738] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9353763a94f34212a8e08d0726c4518e, State: Initialized, Role: FOLLOWER
I20250628 20:03:33.686941 6846 consensus_queue.cc:260] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:33.686941 6845 consensus_queue.cc:260] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:33.687789 6696 heartbeater.cc:499] Master 127.5.100.126:43367 was elected leader, sending a full tablet report...
I20250628 20:03:33.687789 6845 ts_tablet_manager.cc:1428] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250628 20:03:33.687819 6565 heartbeater.cc:499] Master 127.5.100.126:43367 was elected leader, sending a full tablet report...
I20250628 20:03:33.689230 6846 ts_tablet_manager.cc:1428] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250628 20:03:33.689350 6844 ts_tablet_manager.cc:1428] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250628 20:03:33.696297 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:33.697599 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9353763a94f34212a8e08d0726c4518e to finish bootstrapping
I20250628 20:03:33.703425 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b4461325368e4b8dabdd3baaadf17248 to finish bootstrapping
W20250628 20:03:33.704632 6697 tablet.cc:2378] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:33.708724 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0107549ec7014c85819739c61e35466e to finish bootstrapping
I20250628 20:03:33.726251 6781 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5a813da1389c4358b4f85a204062cbe5"
dest_uuid: "0107549ec7014c85819739c61e35466e"
from {username='slave'} at 127.0.0.1:49890
I20250628 20:03:33.726480 6781 raft_consensus.cc:491] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250628 20:03:33.726590 6781 raft_consensus.cc:3058] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:33.727506 6781 raft_consensus.cc:513] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:33.728561 6781 leader_election.cc:290] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [CANDIDATE]: Term 1 election: Requested vote from peers 9353763a94f34212a8e08d0726c4518e (127.5.100.65:44931), b4461325368e4b8dabdd3baaadf17248 (127.5.100.66:33013)
I20250628 20:03:33.732136 5521 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250628 20:03:33.732404 6519 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a813da1389c4358b4f85a204062cbe5" candidate_uuid: "0107549ec7014c85819739c61e35466e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9353763a94f34212a8e08d0726c4518e"
I20250628 20:03:33.732510 6648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a813da1389c4358b4f85a204062cbe5" candidate_uuid: "0107549ec7014c85819739c61e35466e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4461325368e4b8dabdd3baaadf17248"
I20250628 20:03:33.732625 6519 raft_consensus.cc:3058] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:33.732673 6648 raft_consensus.cc:3058] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:33.733829 6648 raft_consensus.cc:2466] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0107549ec7014c85819739c61e35466e in term 1.
I20250628 20:03:33.733829 6519 raft_consensus.cc:2466] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0107549ec7014c85819739c61e35466e in term 1.
I20250628 20:03:33.734251 6716 leader_election.cc:304] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0107549ec7014c85819739c61e35466e, 9353763a94f34212a8e08d0726c4518e; no voters:
I20250628 20:03:33.734504 6850 raft_consensus.cc:2802] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:33.734714 6850 raft_consensus.cc:695] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 1 LEADER]: Becoming Leader. State: Replica: 0107549ec7014c85819739c61e35466e, State: Running, Role: LEADER
I20250628 20:03:33.735065 6850 consensus_queue.cc:237] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:33.738343 6376 catalog_manager.cc:5582] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0107549ec7014c85819739c61e35466e (127.5.100.67). New cstate: current_term: 1 leader_uuid: "0107549ec7014c85819739c61e35466e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } health_report { overall_health: UNKNOWN } } }
W20250628 20:03:33.779247 6566 tablet.cc:2378] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:33.834949 5521 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250628 20:03:33.884054 6828 tablet.cc:2378] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:34.037716 5521 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250628 20:03:34.182792 6850 consensus_queue.cc:1035] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:34.188223 6856 consensus_queue.cc:1035] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:34.478106 6781 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5a813da1389c4358b4f85a204062cbe5"
dest_uuid: "0107549ec7014c85819739c61e35466e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49898
I20250628 20:03:34.478279 6781 raft_consensus.cc:604] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 1 LEADER]: Received request to transfer leadership
I20250628 20:03:34.783618 6883 raft_consensus.cc:991] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e: : Instructing follower 9353763a94f34212a8e08d0726c4518e to start an election
I20250628 20:03:34.783852 6884 raft_consensus.cc:1079] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 1 LEADER]: Signalling peer 9353763a94f34212a8e08d0726c4518e to start an election
I20250628 20:03:34.784459 6519 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5a813da1389c4358b4f85a204062cbe5"
dest_uuid: "9353763a94f34212a8e08d0726c4518e"
from {username='slave'} at 127.5.100.67:42547
I20250628 20:03:34.784646 6519 raft_consensus.cc:491] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250628 20:03:34.784701 6519 raft_consensus.cc:3058] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:34.786041 6519 raft_consensus.cc:513] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:34.787025 6519 leader_election.cc:290] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [CANDIDATE]: Term 2 election: Requested vote from peers 0107549ec7014c85819739c61e35466e (127.5.100.67:34983), b4461325368e4b8dabdd3baaadf17248 (127.5.100.66:33013)
I20250628 20:03:34.791080 6781 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a813da1389c4358b4f85a204062cbe5" candidate_uuid: "9353763a94f34212a8e08d0726c4518e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0107549ec7014c85819739c61e35466e"
I20250628 20:03:34.791255 6781 raft_consensus.cc:3053] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 1 LEADER]: Stepping down as leader of term 1
I20250628 20:03:34.791311 6781 raft_consensus.cc:738] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0107549ec7014c85819739c61e35466e, State: Running, Role: LEADER
I20250628 20:03:34.791347 6648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a813da1389c4358b4f85a204062cbe5" candidate_uuid: "9353763a94f34212a8e08d0726c4518e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b4461325368e4b8dabdd3baaadf17248"
I20250628 20:03:34.791460 6648 raft_consensus.cc:3058] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248 [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:34.791430 6781 consensus_queue.cc:260] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:34.791708 6781 raft_consensus.cc:3058] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:34.792464 6648 raft_consensus.cc:2466] T 5a813da1389c4358b4f85a204062cbe5 P b4461325368e4b8dabdd3baaadf17248 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9353763a94f34212a8e08d0726c4518e in term 2.
I20250628 20:03:34.792594 6781 raft_consensus.cc:2466] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9353763a94f34212a8e08d0726c4518e in term 2.
I20250628 20:03:34.792956 6453 leader_election.cc:304] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0107549ec7014c85819739c61e35466e, 9353763a94f34212a8e08d0726c4518e, b4461325368e4b8dabdd3baaadf17248; no voters:
I20250628 20:03:34.793308 6888 raft_consensus.cc:2802] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 2 FOLLOWER]: Leader election won for term 2
I20250628 20:03:34.793465 6888 raft_consensus.cc:695] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [term 2 LEADER]: Becoming Leader. State: Replica: 9353763a94f34212a8e08d0726c4518e, State: Running, Role: LEADER
I20250628 20:03:34.793777 6888 consensus_queue.cc:237] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } }
I20250628 20:03:34.795892 6377 catalog_manager.cc:5582] T 5a813da1389c4358b4f85a204062cbe5 P 9353763a94f34212a8e08d0726c4518e reported cstate change: term changed from 1 to 2, leader changed from 0107549ec7014c85819739c61e35466e (127.5.100.67) to 9353763a94f34212a8e08d0726c4518e (127.5.100.65). New cstate: current_term: 2 leader_uuid: "9353763a94f34212a8e08d0726c4518e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9353763a94f34212a8e08d0726c4518e" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0107549ec7014c85819739c61e35466e" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4461325368e4b8dabdd3baaadf17248" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33013 } health_report { overall_health: UNKNOWN } } }
I20250628 20:03:35.152093 6781 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5a813da1389c4358b4f85a204062cbe5"
dest_uuid: "0107549ec7014c85819739c61e35466e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49912
I20250628 20:03:35.152235 6781 raft_consensus.cc:604] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 2 FOLLOWER]: Received request to transfer leadership
I20250628 20:03:35.152293 6781 raft_consensus.cc:612] T 5a813da1389c4358b4f85a204062cbe5 P 0107549ec7014c85819739c61e35466e [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250628 20:03:35.160743 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 6437
I20250628 20:03:35.166452 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 6568
I20250628 20:03:35.171733 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 6699
I20250628 20:03:35.176798 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 6346
2025-06-28T20:03:35Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (2337 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250628 20:03:35.196568 5521 test_util.cc:276] Using random seed: 1244238739
I20250628 20:03:35.197301 5521 ts_itest-base.cc:115] Starting cluster with:
I20250628 20:03:35.197347 5521 ts_itest-base.cc:116] --------------
I20250628 20:03:35.197358 5521 ts_itest-base.cc:117] 3 tablet servers
I20250628 20:03:35.197367 5521 ts_itest-base.cc:118] 3 replicas per TS
I20250628 20:03:35.197376 5521 ts_itest-base.cc:119] --------------
2025-06-28T20:03:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T20:03:35Z Disabled control of system clock
I20250628 20:03:35.204710 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:36429
--webserver_interface=127.5.100.126
--webserver_port=0
--builtin_ntp_servers=127.5.100.84:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:36429 with env {}
W20250628 20:03:35.308622 6917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:35.308887 6917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:35.308944 6917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:35.312220 6917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:35.312300 6917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:35.312328 6917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:35.312351 6917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:35.316177 6917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:36429
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:36429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:35.317348 6917 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:35.318573 6917 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:35.325196 6925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:35.325255 6922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:35.325397 6917 server_base.cc:1048] running on GCE node
W20250628 20:03:35.325210 6923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:35.326028 6917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:35.326943 6917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:35.328092 6917 hybrid_clock.cc:648] HybridClock initialized: now 1751141015328086 us; error 33 us; skew 500 ppm
I20250628 20:03:35.330024 6917 webserver.cc:469] Webserver started at http://127.5.100.126:38107/ using document root <none> and password file <none>
I20250628 20:03:35.330582 6917 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:35.330641 6917 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:35.330814 6917 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:35.332466 6917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "83703e50d05b421092a9eb75598b5612"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.332947 6917 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "83703e50d05b421092a9eb75598b5612"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.336371 6917 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:35.338640 6931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.339672 6917 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250628 20:03:35.339787 6917 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "83703e50d05b421092a9eb75598b5612"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.339874 6917 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:35.351725 6917 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:35.352403 6917 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:35.352569 6917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:35.359462 6917 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:36429
I20250628 20:03:35.359470 6983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:36429 every 8 connection(s)
I20250628 20:03:35.360466 6917 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:35.363099 6984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:35.368191 6984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612: Bootstrap starting.
I20250628 20:03:35.369411 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 6917
I20250628 20:03:35.369526 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 20:03:35.370805 6984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:35.371681 6984 log.cc:826] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:35.373659 6984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612: No bootstrap required, opened a new log
I20250628 20:03:35.376632 6984 raft_consensus.cc:357] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83703e50d05b421092a9eb75598b5612" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 36429 } }
I20250628 20:03:35.376888 6984 raft_consensus.cc:383] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:35.376950 6984 raft_consensus.cc:738] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83703e50d05b421092a9eb75598b5612, State: Initialized, Role: FOLLOWER
I20250628 20:03:35.377367 6984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83703e50d05b421092a9eb75598b5612" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 36429 } }
I20250628 20:03:35.377485 6984 raft_consensus.cc:397] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:35.377521 6984 raft_consensus.cc:491] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:35.377600 6984 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:35.378352 6984 raft_consensus.cc:513] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83703e50d05b421092a9eb75598b5612" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 36429 } }
I20250628 20:03:35.378708 6984 leader_election.cc:304] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83703e50d05b421092a9eb75598b5612; no voters:
I20250628 20:03:35.379016 6984 leader_election.cc:290] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:35.379092 6989 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:35.379289 6989 raft_consensus.cc:695] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [term 1 LEADER]: Becoming Leader. State: Replica: 83703e50d05b421092a9eb75598b5612, State: Running, Role: LEADER
I20250628 20:03:35.379602 6989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83703e50d05b421092a9eb75598b5612" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 36429 } }
I20250628 20:03:35.380146 6984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:35.380995 6990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83703e50d05b421092a9eb75598b5612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83703e50d05b421092a9eb75598b5612" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 36429 } } }
I20250628 20:03:35.381114 6990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:35.381289 6991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83703e50d05b421092a9eb75598b5612. Latest consensus state: current_term: 1 leader_uuid: "83703e50d05b421092a9eb75598b5612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83703e50d05b421092a9eb75598b5612" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 36429 } } }
I20250628 20:03:35.381444 6991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:35.381711 6998 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:35.384266 6998 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:35.389155 6998 catalog_manager.cc:1349] Generated new cluster ID: 9f89568040724af6947414c9d443fa38
I20250628 20:03:35.389232 6998 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:35.406754 6998 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:35.407693 6998 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:35.414472 6998 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 83703e50d05b421092a9eb75598b5612: Generated new TSK 0
I20250628 20:03:35.415200 6998 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:35.421979 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:0
--local_ip_for_outbound_sockets=127.5.100.65
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:36429
--builtin_ntp_servers=127.5.100.84:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 20:03:35.527448 7008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:35.527673 7008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:35.527715 7008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:35.531054 7008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:35.531173 7008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:35.534977 7008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:36429
--never_fsync=true
--heap_profile_path=/tmp/kudu.7008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:35.536087 7008 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:35.537312 7008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:35.544966 7016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:35.544935 7014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:35.544987 7013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:35.545073 7008 server_base.cc:1048] running on GCE node
I20250628 20:03:35.545579 7008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:35.546205 7008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:35.547355 7008 hybrid_clock.cc:648] HybridClock initialized: now 1751141015547339 us; error 29 us; skew 500 ppm
I20250628 20:03:35.549305 7008 webserver.cc:469] Webserver started at http://127.5.100.65:33665/ using document root <none> and password file <none>
I20250628 20:03:35.549934 7008 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:35.549993 7008 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:35.550175 7008 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:35.551786 7008 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ccae2cdc864d40e983624cdff89c706c"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.552276 7008 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ccae2cdc864d40e983624cdff89c706c"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.555689 7008 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:35.557955 7022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.558969 7008 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:35.559077 7008 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ccae2cdc864d40e983624cdff89c706c"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.559167 7008 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:35.604384 7008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:35.605079 7008 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:35.605257 7008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:35.605861 7008 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:35.606823 7008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:35.606873 7008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.606931 7008 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:35.606956 7008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.615715 7008 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:33849
I20250628 20:03:35.615733 7135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:33849 every 8 connection(s)
I20250628 20:03:35.616707 7008 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 20:03:35.617806 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 7008
I20250628 20:03:35.617902 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 20:03:35.619891 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:0
--local_ip_for_outbound_sockets=127.5.100.66
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:36429
--builtin_ntp_servers=127.5.100.84:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:35.626042 7136 heartbeater.cc:344] Connected to a master server at 127.5.100.126:36429
I20250628 20:03:35.626317 7136 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:35.626890 7136 heartbeater.cc:507] Master 127.5.100.126:36429 requested a full tablet report, sending...
I20250628 20:03:35.628355 6948 ts_manager.cc:194] Registered new tserver with Master: ccae2cdc864d40e983624cdff89c706c (127.5.100.65:33849)
I20250628 20:03:35.629514 6948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:54701
W20250628 20:03:35.723348 7139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:35.723568 7139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:35.723618 7139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:35.726826 7139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:35.726958 7139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:35.730688 7139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:36429
--never_fsync=true
--heap_profile_path=/tmp/kudu.7139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:35.731796 7139 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:35.732954 7139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:35.740422 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:35.740443 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:35.740422 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:35.740768 7139 server_base.cc:1048] running on GCE node
I20250628 20:03:35.741106 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:35.741647 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:35.742810 7139 hybrid_clock.cc:648] HybridClock initialized: now 1751141015742793 us; error 30 us; skew 500 ppm
I20250628 20:03:35.744686 7139 webserver.cc:469] Webserver started at http://127.5.100.66:46573/ using document root <none> and password file <none>
I20250628 20:03:35.745251 7139 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:35.745309 7139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:35.745494 7139 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:35.747095 7139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fcc0bbc96599403fa15c7926caa4d75b"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.747539 7139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fcc0bbc96599403fa15c7926caa4d75b"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.750805 7139 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250628 20:03:35.752883 7153 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.753896 7139 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250628 20:03:35.754006 7139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fcc0bbc96599403fa15c7926caa4d75b"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.754093 7139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:35.774874 7139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:35.775588 7139 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:35.775767 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:35.776350 7139 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:35.777340 7139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:35.777388 7139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.777441 7139 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:35.777469 7139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.786132 7139 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:38661
I20250628 20:03:35.786180 7266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:38661 every 8 connection(s)
I20250628 20:03:35.787098 7139 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 20:03:35.795462 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 7139
I20250628 20:03:35.795567 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 20:03:35.796041 7267 heartbeater.cc:344] Connected to a master server at 127.5.100.126:36429
I20250628 20:03:35.796309 7267 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:35.796929 7267 heartbeater.cc:507] Master 127.5.100.126:36429 requested a full tablet report, sending...
I20250628 20:03:35.797523 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:0
--local_ip_for_outbound_sockets=127.5.100.67
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:36429
--builtin_ntp_servers=127.5.100.84:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:35.797935 6948 ts_manager.cc:194] Registered new tserver with Master: fcc0bbc96599403fa15c7926caa4d75b (127.5.100.66:38661)
I20250628 20:03:35.798475 6948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:52005
W20250628 20:03:35.905570 7270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:35.905850 7270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:35.905900 7270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:35.909137 7270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:35.909267 7270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:35.913123 7270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:36429
--never_fsync=true
--heap_profile_path=/tmp/kudu.7270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:35.914355 7270 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:35.915514 7270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:35.923305 7275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:35.923449 7278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:35.923354 7276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:35.923348 7270 server_base.cc:1048] running on GCE node
I20250628 20:03:35.924091 7270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:35.924693 7270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:35.925832 7270 hybrid_clock.cc:648] HybridClock initialized: now 1751141015925816 us; error 31 us; skew 500 ppm
I20250628 20:03:35.927680 7270 webserver.cc:469] Webserver started at http://127.5.100.67:42577/ using document root <none> and password file <none>
I20250628 20:03:35.928256 7270 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:35.928313 7270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:35.928490 7270 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:35.930109 7270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9c90f61b5e0b4981baf985d2b0356f74"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.930573 7270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9c90f61b5e0b4981baf985d2b0356f74"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.933900 7270 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:35.936082 7284 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.937103 7270 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:35.937213 7270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9c90f61b5e0b4981baf985d2b0356f74"
format_stamp: "Formatted at 2025-06-28 20:03:35 on dist-test-slave-rmcj"
I20250628 20:03:35.937301 7270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:35.952407 7270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:35.953111 7270 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:35.953310 7270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:35.953908 7270 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:35.954845 7270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:35.954895 7270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.954947 7270 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:35.954983 7270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:35.963594 7270 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:45229
I20250628 20:03:35.963610 7397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:45229 every 8 connection(s)
I20250628 20:03:35.964614 7270 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 20:03:35.973631 7398 heartbeater.cc:344] Connected to a master server at 127.5.100.126:36429
I20250628 20:03:35.973824 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 7270
I20250628 20:03:35.973923 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 20:03:35.973937 7398 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:35.974520 7398 heartbeater.cc:507] Master 127.5.100.126:36429 requested a full tablet report, sending...
I20250628 20:03:35.975632 6948 ts_manager.cc:194] Registered new tserver with Master: 9c90f61b5e0b4981baf985d2b0356f74 (127.5.100.67:45229)
I20250628 20:03:35.976258 6948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:52351
I20250628 20:03:35.987886 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:35.995805 6948 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34148:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 20:03:35.998274 6948 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 20:03:36.013957 7332 tablet_service.cc:1468] Processing CreateTablet for tablet 77b6d041cfa54b6d99cf499bc0818f0a (DEFAULT_TABLE table=TestTable [id=17dcb751e3d14a09a5dfd7e5d21b6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:36.013969 7070 tablet_service.cc:1468] Processing CreateTablet for tablet 77b6d041cfa54b6d99cf499bc0818f0a (DEFAULT_TABLE table=TestTable [id=17dcb751e3d14a09a5dfd7e5d21b6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:36.014235 7199 tablet_service.cc:1468] Processing CreateTablet for tablet 77b6d041cfa54b6d99cf499bc0818f0a (DEFAULT_TABLE table=TestTable [id=17dcb751e3d14a09a5dfd7e5d21b6d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:36.014845 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b6d041cfa54b6d99cf499bc0818f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:36.014860 7070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b6d041cfa54b6d99cf499bc0818f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:36.014904 7199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77b6d041cfa54b6d99cf499bc0818f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:36.019474 7415 tablet_bootstrap.cc:492] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74: Bootstrap starting.
I20250628 20:03:36.020910 7416 tablet_bootstrap.cc:492] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b: Bootstrap starting.
I20250628 20:03:36.021119 7417 tablet_bootstrap.cc:492] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c: Bootstrap starting.
I20250628 20:03:36.021781 7415 tablet_bootstrap.cc:654] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:36.022444 7415 log.cc:826] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:36.022959 7416 tablet_bootstrap.cc:654] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:36.022960 7417 tablet_bootstrap.cc:654] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:36.023716 7417 log.cc:826] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:36.023716 7416 log.cc:826] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:36.023905 7415 tablet_bootstrap.cc:492] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74: No bootstrap required, opened a new log
I20250628 20:03:36.024078 7415 ts_tablet_manager.cc:1397] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250628 20:03:36.025437 7416 tablet_bootstrap.cc:492] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b: No bootstrap required, opened a new log
I20250628 20:03:36.025570 7417 tablet_bootstrap.cc:492] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c: No bootstrap required, opened a new log
I20250628 20:03:36.025611 7416 ts_tablet_manager.cc:1397] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250628 20:03:36.025748 7417 ts_tablet_manager.cc:1397] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250628 20:03:36.026688 7415 raft_consensus.cc:357] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
I20250628 20:03:36.026876 7415 raft_consensus.cc:383] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:36.026929 7415 raft_consensus.cc:738] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c90f61b5e0b4981baf985d2b0356f74, State: Initialized, Role: FOLLOWER
I20250628 20:03:36.027314 7415 consensus_queue.cc:260] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
I20250628 20:03:36.027947 7398 heartbeater.cc:499] Master 127.5.100.126:36429 was elected leader, sending a full tablet report...
I20250628 20:03:36.028115 7415 ts_tablet_manager.cc:1428] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250628 20:03:36.028446 7417 raft_consensus.cc:357] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
I20250628 20:03:36.028641 7417 raft_consensus.cc:383] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:36.028694 7417 raft_consensus.cc:738] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccae2cdc864d40e983624cdff89c706c, State: Initialized, Role: FOLLOWER
I20250628 20:03:36.028926 7416 raft_consensus.cc:357] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
I20250628 20:03:36.029114 7416 raft_consensus.cc:383] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:36.029163 7416 raft_consensus.cc:738] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc0bbc96599403fa15c7926caa4d75b, State: Initialized, Role: FOLLOWER
I20250628 20:03:36.029106 7417 consensus_queue.cc:260] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
I20250628 20:03:36.029589 7416 consensus_queue.cc:260] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
I20250628 20:03:36.029953 7417 ts_tablet_manager.cc:1428] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 20:03:36.030282 7136 heartbeater.cc:499] Master 127.5.100.126:36429 was elected leader, sending a full tablet report...
I20250628 20:03:36.030390 7416 ts_tablet_manager.cc:1428] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250628 20:03:36.030733 7267 heartbeater.cc:499] Master 127.5.100.126:36429 was elected leader, sending a full tablet report...
W20250628 20:03:36.037919 7268 tablet.cc:2378] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:36.105610 7421 raft_consensus.cc:491] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:36.105837 7421 raft_consensus.cc:513] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
I20250628 20:03:36.106915 7421 leader_election.cc:290] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccae2cdc864d40e983624cdff89c706c (127.5.100.65:33849), fcc0bbc96599403fa15c7926caa4d75b (127.5.100.66:38661)
I20250628 20:03:36.110468 7090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77b6d041cfa54b6d99cf499bc0818f0a" candidate_uuid: "9c90f61b5e0b4981baf985d2b0356f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccae2cdc864d40e983624cdff89c706c" is_pre_election: true
I20250628 20:03:36.110736 7090 raft_consensus.cc:2466] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c90f61b5e0b4981baf985d2b0356f74 in term 0.
I20250628 20:03:36.111205 7286 leader_election.cc:304] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c90f61b5e0b4981baf985d2b0356f74, ccae2cdc864d40e983624cdff89c706c; no voters:
I20250628 20:03:36.111271 7221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77b6d041cfa54b6d99cf499bc0818f0a" candidate_uuid: "9c90f61b5e0b4981baf985d2b0356f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc0bbc96599403fa15c7926caa4d75b" is_pre_election: true
I20250628 20:03:36.111455 7421 raft_consensus.cc:2802] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 20:03:36.111514 7421 raft_consensus.cc:491] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 20:03:36.111554 7421 raft_consensus.cc:3058] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:36.111531 7221 raft_consensus.cc:2466] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c90f61b5e0b4981baf985d2b0356f74 in term 0.
I20250628 20:03:36.112484 7421 raft_consensus.cc:513] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
I20250628 20:03:36.112809 7421 leader_election.cc:290] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [CANDIDATE]: Term 1 election: Requested vote from peers ccae2cdc864d40e983624cdff89c706c (127.5.100.65:33849), fcc0bbc96599403fa15c7926caa4d75b (127.5.100.66:38661)
I20250628 20:03:36.113142 7221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77b6d041cfa54b6d99cf499bc0818f0a" candidate_uuid: "9c90f61b5e0b4981baf985d2b0356f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc0bbc96599403fa15c7926caa4d75b"
I20250628 20:03:36.113121 7090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77b6d041cfa54b6d99cf499bc0818f0a" candidate_uuid: "9c90f61b5e0b4981baf985d2b0356f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccae2cdc864d40e983624cdff89c706c"
I20250628 20:03:36.113260 7090 raft_consensus.cc:3058] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:36.113257 7221 raft_consensus.cc:3058] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:36.114370 7221 raft_consensus.cc:2466] T 77b6d041cfa54b6d99cf499bc0818f0a P fcc0bbc96599403fa15c7926caa4d75b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c90f61b5e0b4981baf985d2b0356f74 in term 1.
I20250628 20:03:36.114666 7287 leader_election.cc:304] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c90f61b5e0b4981baf985d2b0356f74, fcc0bbc96599403fa15c7926caa4d75b; no voters:
I20250628 20:03:36.114738 7090 raft_consensus.cc:2466] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c90f61b5e0b4981baf985d2b0356f74 in term 1.
I20250628 20:03:36.114851 7421 raft_consensus.cc:2802] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:36.115082 7421 raft_consensus.cc:695] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [term 1 LEADER]: Becoming Leader. State: Replica: 9c90f61b5e0b4981baf985d2b0356f74, State: Running, Role: LEADER
I20250628 20:03:36.115406 7421 consensus_queue.cc:237] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } }
W20250628 20:03:36.117790 7137 tablet.cc:2378] T 77b6d041cfa54b6d99cf499bc0818f0a P ccae2cdc864d40e983624cdff89c706c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:36.118371 6947 catalog_manager.cc:5582] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c90f61b5e0b4981baf985d2b0356f74 (127.5.100.67). New cstate: current_term: 1 leader_uuid: "9c90f61b5e0b4981baf985d2b0356f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } health_report { overall_health: UNKNOWN } } }
I20250628 20:03:36.137248 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:36.138679 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ccae2cdc864d40e983624cdff89c706c to finish bootstrapping
I20250628 20:03:36.143658 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fcc0bbc96599403fa15c7926caa4d75b to finish bootstrapping
I20250628 20:03:36.148679 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9c90f61b5e0b4981baf985d2b0356f74 to finish bootstrapping
I20250628 20:03:36.154596 6947 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34148:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250628 20:03:36.155059 6947 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 20:03:36.159534 7070 tablet_service.cc:1468] Processing CreateTablet for tablet 2140016164fd4f57a30b931400441725 (DEFAULT_TABLE table=TestAnotherTable [id=040f35d0b7db4feda8ac23a0a9fd96f0]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 20:03:36.159646 7199 tablet_service.cc:1468] Processing CreateTablet for tablet 2140016164fd4f57a30b931400441725 (DEFAULT_TABLE table=TestAnotherTable [id=040f35d0b7db4feda8ac23a0a9fd96f0]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 20:03:36.159703 7332 tablet_service.cc:1468] Processing CreateTablet for tablet 2140016164fd4f57a30b931400441725 (DEFAULT_TABLE table=TestAnotherTable [id=040f35d0b7db4feda8ac23a0a9fd96f0]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 20:03:36.159878 7070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2140016164fd4f57a30b931400441725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:36.159922 7199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2140016164fd4f57a30b931400441725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:36.159945 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2140016164fd4f57a30b931400441725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:36.161530 7416 tablet_bootstrap.cc:492] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b: Bootstrap starting.
I20250628 20:03:36.161906 7415 tablet_bootstrap.cc:492] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74: Bootstrap starting.
I20250628 20:03:36.162569 7416 tablet_bootstrap.cc:654] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:36.162838 7415 tablet_bootstrap.cc:654] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:36.163934 7415 tablet_bootstrap.cc:492] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74: No bootstrap required, opened a new log
I20250628 20:03:36.164018 7415 ts_tablet_manager.cc:1397] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 20:03:36.164040 7416 tablet_bootstrap.cc:492] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b: No bootstrap required, opened a new log
I20250628 20:03:36.164129 7416 ts_tablet_manager.cc:1397] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250628 20:03:36.164499 7415 raft_consensus.cc:357] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.164592 7415 raft_consensus.cc:383] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:36.164623 7415 raft_consensus.cc:738] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c90f61b5e0b4981baf985d2b0356f74, State: Initialized, Role: FOLLOWER
I20250628 20:03:36.164583 7416 raft_consensus.cc:357] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.164670 7416 raft_consensus.cc:383] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:36.164710 7416 raft_consensus.cc:738] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc0bbc96599403fa15c7926caa4d75b, State: Initialized, Role: FOLLOWER
I20250628 20:03:36.164759 7415 consensus_queue.cc:260] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.164821 7416 consensus_queue.cc:260] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.165009 7415 ts_tablet_manager.cc:1428] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 20:03:36.165048 7416 ts_tablet_manager.cc:1428] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 20:03:36.165525 7417 tablet_bootstrap.cc:492] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c: Bootstrap starting.
I20250628 20:03:36.166543 7417 tablet_bootstrap.cc:654] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:36.167564 7417 tablet_bootstrap.cc:492] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c: No bootstrap required, opened a new log
I20250628 20:03:36.167642 7417 ts_tablet_manager.cc:1397] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:36.168108 7417 raft_consensus.cc:357] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.168200 7417 raft_consensus.cc:383] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:36.168228 7417 raft_consensus.cc:738] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccae2cdc864d40e983624cdff89c706c, State: Initialized, Role: FOLLOWER
I20250628 20:03:36.168350 7417 consensus_queue.cc:260] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.168609 7417 ts_tablet_manager.cc:1428] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250628 20:03:36.215446 7399 tablet.cc:2378] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:36.391209 7421 raft_consensus.cc:491] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:36.391355 7421 raft_consensus.cc:513] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.391794 7421 leader_election.cc:290] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccae2cdc864d40e983624cdff89c706c (127.5.100.65:33849), fcc0bbc96599403fa15c7926caa4d75b (127.5.100.66:38661)
I20250628 20:03:36.392277 7221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2140016164fd4f57a30b931400441725" candidate_uuid: "9c90f61b5e0b4981baf985d2b0356f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc0bbc96599403fa15c7926caa4d75b" is_pre_election: true
I20250628 20:03:36.392287 7090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2140016164fd4f57a30b931400441725" candidate_uuid: "9c90f61b5e0b4981baf985d2b0356f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccae2cdc864d40e983624cdff89c706c" is_pre_election: true
I20250628 20:03:36.392414 7221 raft_consensus.cc:2466] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c90f61b5e0b4981baf985d2b0356f74 in term 0.
I20250628 20:03:36.392426 7090 raft_consensus.cc:2466] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c90f61b5e0b4981baf985d2b0356f74 in term 0.
I20250628 20:03:36.392711 7287 leader_election.cc:304] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c90f61b5e0b4981baf985d2b0356f74, fcc0bbc96599403fa15c7926caa4d75b; no voters:
I20250628 20:03:36.392874 7421 raft_consensus.cc:2802] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 20:03:36.392940 7421 raft_consensus.cc:491] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 20:03:36.392969 7421 raft_consensus.cc:3058] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:36.393775 7421 raft_consensus.cc:513] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.394088 7421 leader_election.cc:290] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [CANDIDATE]: Term 1 election: Requested vote from peers ccae2cdc864d40e983624cdff89c706c (127.5.100.65:33849), fcc0bbc96599403fa15c7926caa4d75b (127.5.100.66:38661)
I20250628 20:03:36.394459 7090 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2140016164fd4f57a30b931400441725" candidate_uuid: "9c90f61b5e0b4981baf985d2b0356f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccae2cdc864d40e983624cdff89c706c"
I20250628 20:03:36.394555 7090 raft_consensus.cc:3058] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:36.395284 7090 raft_consensus.cc:2466] T 2140016164fd4f57a30b931400441725 P ccae2cdc864d40e983624cdff89c706c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c90f61b5e0b4981baf985d2b0356f74 in term 1.
I20250628 20:03:36.395522 7221 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2140016164fd4f57a30b931400441725" candidate_uuid: "9c90f61b5e0b4981baf985d2b0356f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc0bbc96599403fa15c7926caa4d75b"
I20250628 20:03:36.395540 7286 leader_election.cc:304] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c90f61b5e0b4981baf985d2b0356f74, ccae2cdc864d40e983624cdff89c706c; no voters:
I20250628 20:03:36.395633 7221 raft_consensus.cc:3058] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:36.395686 7421 raft_consensus.cc:2802] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:36.395773 7421 raft_consensus.cc:695] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [term 1 LEADER]: Becoming Leader. State: Replica: 9c90f61b5e0b4981baf985d2b0356f74, State: Running, Role: LEADER
I20250628 20:03:36.395918 7421 consensus_queue.cc:237] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } }
I20250628 20:03:36.396595 7221 raft_consensus.cc:2466] T 2140016164fd4f57a30b931400441725 P fcc0bbc96599403fa15c7926caa4d75b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c90f61b5e0b4981baf985d2b0356f74 in term 1.
I20250628 20:03:36.397655 6945 catalog_manager.cc:5582] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c90f61b5e0b4981baf985d2b0356f74 (127.5.100.67). New cstate: current_term: 1 leader_uuid: "9c90f61b5e0b4981baf985d2b0356f74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c90f61b5e0b4981baf985d2b0356f74" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45229 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:36.515265 7421 consensus_queue.cc:1035] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:36.520586 7425 consensus_queue.cc:1035] T 77b6d041cfa54b6d99cf499bc0818f0a P 9c90f61b5e0b4981baf985d2b0356f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250628 20:03:36.564016 7432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:36.564320 7432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:36.568377 7432 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 20:03:36.709957 7449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:36.710204 7449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:36.713341 7449 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250628 20:03:36.800246 7425 consensus_queue.cc:1035] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccae2cdc864d40e983624cdff89c706c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:36.803429 7425 consensus_queue.cc:1035] T 2140016164fd4f57a30b931400441725 P 9c90f61b5e0b4981baf985d2b0356f74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcc0bbc96599403fa15c7926caa4d75b" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250628 20:03:36.853072 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:36.853329 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:36.856555 7459 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 20:03:36.993472 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:36.993743 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:36.996946 7471 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250628 20:03:37.030483 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 7008
I20250628 20:03:37.035732 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 7139
I20250628 20:03:37.040509 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 7270
I20250628 20:03:37.045275 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 6917
2025-06-28T20:03:37Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (1868 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250628 20:03:37.065099 5521 test_util.cc:276] Using random seed: 1246107267
I20250628 20:03:37.065866 5521 ts_itest-base.cc:115] Starting cluster with:
I20250628 20:03:37.065917 5521 ts_itest-base.cc:116] --------------
I20250628 20:03:37.065927 5521 ts_itest-base.cc:117] 3 tablet servers
I20250628 20:03:37.065936 5521 ts_itest-base.cc:118] 3 replicas per TS
I20250628 20:03:37.065948 5521 ts_itest-base.cc:119] --------------
2025-06-28T20:03:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T20:03:37Z Disabled control of system clock
I20250628 20:03:37.073477 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:45935
--webserver_interface=127.5.100.126
--webserver_port=0
--builtin_ntp_servers=127.5.100.84:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:45935
--superuser_acl=no-such-user with env {}
W20250628 20:03:37.176223 7488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:37.176460 7488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:37.176522 7488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:37.179693 7488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:37.179751 7488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:37.179778 7488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:37.179811 7488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:37.183535 7488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:45935
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:37.184625 7488 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:37.185775 7488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:37.192437 7493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:37.192613 7488 server_base.cc:1048] running on GCE node
W20250628 20:03:37.192526 7494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:37.192487 7496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:37.193176 7488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:37.194118 7488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:37.195258 7488 hybrid_clock.cc:648] HybridClock initialized: now 1751141017195243 us; error 29 us; skew 500 ppm
I20250628 20:03:37.196969 7488 webserver.cc:469] Webserver started at http://127.5.100.126:35765/ using document root <none> and password file <none>
I20250628 20:03:37.197510 7488 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:37.197566 7488 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:37.197769 7488 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:37.199404 7488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "62392ce09cbe4ca8b313b5c70de24e72"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.199844 7488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "62392ce09cbe4ca8b313b5c70de24e72"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.203001 7488 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250628 20:03:37.204998 7502 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.205992 7488 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:37.206115 7488 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "62392ce09cbe4ca8b313b5c70de24e72"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.206198 7488 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:37.239378 7488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:37.240074 7488 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:37.240257 7488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:37.247013 7488 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:45935
I20250628 20:03:37.247022 7554 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:45935 every 8 connection(s)
I20250628 20:03:37.248041 7488 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:37.248364 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 7488
I20250628 20:03:37.248459 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 20:03:37.250851 7555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:37.256325 7555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72: Bootstrap starting.
I20250628 20:03:37.258476 7555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:37.259156 7555 log.cc:826] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:37.260802 7555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72: No bootstrap required, opened a new log
I20250628 20:03:37.263324 7555 raft_consensus.cc:357] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62392ce09cbe4ca8b313b5c70de24e72" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 45935 } }
I20250628 20:03:37.263578 7555 raft_consensus.cc:383] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:37.263643 7555 raft_consensus.cc:738] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62392ce09cbe4ca8b313b5c70de24e72, State: Initialized, Role: FOLLOWER
I20250628 20:03:37.264111 7555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62392ce09cbe4ca8b313b5c70de24e72" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 45935 } }
I20250628 20:03:37.264268 7555 raft_consensus.cc:397] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:37.264333 7555 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:37.264438 7555 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:37.265234 7555 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62392ce09cbe4ca8b313b5c70de24e72" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 45935 } }
I20250628 20:03:37.265553 7555 leader_election.cc:304] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62392ce09cbe4ca8b313b5c70de24e72; no voters:
I20250628 20:03:37.265870 7555 leader_election.cc:290] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:37.266008 7560 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:37.266243 7560 raft_consensus.cc:695] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [term 1 LEADER]: Becoming Leader. State: Replica: 62392ce09cbe4ca8b313b5c70de24e72, State: Running, Role: LEADER
I20250628 20:03:37.266556 7560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62392ce09cbe4ca8b313b5c70de24e72" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 45935 } }
I20250628 20:03:37.266971 7555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:37.268266 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62392ce09cbe4ca8b313b5c70de24e72. Latest consensus state: current_term: 1 leader_uuid: "62392ce09cbe4ca8b313b5c70de24e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62392ce09cbe4ca8b313b5c70de24e72" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 45935 } } }
I20250628 20:03:37.268393 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:37.268697 7561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62392ce09cbe4ca8b313b5c70de24e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62392ce09cbe4ca8b313b5c70de24e72" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 45935 } } }
I20250628 20:03:37.268855 7561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:37.269081 7569 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:37.271752 7569 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:37.276638 7569 catalog_manager.cc:1349] Generated new cluster ID: ef114cb61a944511b7146f28fdce03bd
I20250628 20:03:37.276726 7569 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:37.299250 7569 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:37.300215 7569 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:37.306252 7569 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 62392ce09cbe4ca8b313b5c70de24e72: Generated new TSK 0
I20250628 20:03:37.306951 7569 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:37.316502 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:0
--local_ip_for_outbound_sockets=127.5.100.65
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:45935
--builtin_ntp_servers=127.5.100.84:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 20:03:37.422158 7579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:37.422366 7579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:37.422412 7579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:37.425596 7579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:37.425732 7579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:37.429471 7579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:45935
--never_fsync=true
--heap_profile_path=/tmp/kudu.7579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:37.430611 7579 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:37.431740 7579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:37.439253 7584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:37.439355 7579 server_base.cc:1048] running on GCE node
W20250628 20:03:37.439435 7585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:37.439566 7587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:37.439915 7579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:37.440467 7579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:37.441609 7579 hybrid_clock.cc:648] HybridClock initialized: now 1751141017441592 us; error 31 us; skew 500 ppm
I20250628 20:03:37.443434 7579 webserver.cc:469] Webserver started at http://127.5.100.65:36809/ using document root <none> and password file <none>
I20250628 20:03:37.443995 7579 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:37.444051 7579 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:37.444242 7579 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:37.445878 7579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "527f6b9bdbf544c2935853102b9488c2"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.446348 7579 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "527f6b9bdbf544c2935853102b9488c2"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.449541 7579 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:37.451733 7593 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.452755 7579 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:37.452863 7579 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "527f6b9bdbf544c2935853102b9488c2"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.452951 7579 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:37.470125 7579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:37.470788 7579 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:37.470968 7579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:37.471534 7579 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:37.472537 7579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:37.472587 7579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.472628 7579 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:37.472651 7579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.481122 7579 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:33837
I20250628 20:03:37.481144 7706 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:33837 every 8 connection(s)
I20250628 20:03:37.482172 7579 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 20:03:37.491042 7707 heartbeater.cc:344] Connected to a master server at 127.5.100.126:45935
I20250628 20:03:37.491315 7707 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:37.491890 7707 heartbeater.cc:507] Master 127.5.100.126:45935 requested a full tablet report, sending...
I20250628 20:03:37.492415 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 7579
I20250628 20:03:37.492509 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 20:03:37.493417 7519 ts_manager.cc:194] Registered new tserver with Master: 527f6b9bdbf544c2935853102b9488c2 (127.5.100.65:33837)
I20250628 20:03:37.494654 7519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:44411
I20250628 20:03:37.494769 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:0
--local_ip_for_outbound_sockets=127.5.100.66
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:45935
--builtin_ntp_servers=127.5.100.84:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 20:03:37.597546 7710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:37.597777 7710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:37.597836 7710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:37.600906 7710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:37.601009 7710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:37.604550 7710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:45935
--never_fsync=true
--heap_profile_path=/tmp/kudu.7710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:37.605594 7710 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:37.606787 7710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:37.614439 7715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:37.614560 7710 server_base.cc:1048] running on GCE node
W20250628 20:03:37.614492 7718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:37.614416 7716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:37.615036 7710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:37.615604 7710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:37.616747 7710 hybrid_clock.cc:648] HybridClock initialized: now 1751141017616732 us; error 32 us; skew 500 ppm
I20250628 20:03:37.618590 7710 webserver.cc:469] Webserver started at http://127.5.100.66:35627/ using document root <none> and password file <none>
I20250628 20:03:37.619165 7710 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:37.619220 7710 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:37.619400 7710 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:37.620968 7710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "32650294f7f24aa0ad7cb240326417a0"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.621421 7710 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "32650294f7f24aa0ad7cb240326417a0"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.624806 7710 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:37.627054 7724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.628077 7710 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250628 20:03:37.628198 7710 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "32650294f7f24aa0ad7cb240326417a0"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.628290 7710 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:37.641470 7710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:37.642172 7710 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:37.642338 7710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:37.642884 7710 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:37.643805 7710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:37.643854 7710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.643908 7710 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:37.643931 7710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.652567 7710 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:33477
I20250628 20:03:37.652588 7837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:33477 every 8 connection(s)
I20250628 20:03:37.653517 7710 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 20:03:37.660631 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 7710
I20250628 20:03:37.660725 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 20:03:37.662473 7838 heartbeater.cc:344] Connected to a master server at 127.5.100.126:45935
I20250628 20:03:37.662720 7838 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:37.662866 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:0
--local_ip_for_outbound_sockets=127.5.100.67
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:45935
--builtin_ntp_servers=127.5.100.84:35443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:37.663293 7838 heartbeater.cc:507] Master 127.5.100.126:45935 requested a full tablet report, sending...
I20250628 20:03:37.664335 7519 ts_manager.cc:194] Registered new tserver with Master: 32650294f7f24aa0ad7cb240326417a0 (127.5.100.66:33477)
I20250628 20:03:37.664952 7519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:47825
W20250628 20:03:37.764966 7841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:37.765179 7841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:37.765221 7841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:37.768358 7841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:37.768481 7841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:37.772054 7841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:35443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:45935
--never_fsync=true
--heap_profile_path=/tmp/kudu.7841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:37.773139 7841 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:37.774259 7841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:37.781533 7849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:37.781555 7846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:37.781665 7841 server_base.cc:1048] running on GCE node
W20250628 20:03:37.781605 7847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:37.782258 7841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:37.782786 7841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:37.783938 7841 hybrid_clock.cc:648] HybridClock initialized: now 1751141017783923 us; error 25 us; skew 500 ppm
I20250628 20:03:37.785763 7841 webserver.cc:469] Webserver started at http://127.5.100.67:33627/ using document root <none> and password file <none>
I20250628 20:03:37.786329 7841 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:37.786383 7841 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:37.786566 7841 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:37.788174 7841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "936c980484cb48568ec596a4127803e4"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.788614 7841 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "936c980484cb48568ec596a4127803e4"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.791847 7841 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:37.793895 7855 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.794884 7841 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:37.794996 7841 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "936c980484cb48568ec596a4127803e4"
format_stamp: "Formatted at 2025-06-28 20:03:37 on dist-test-slave-rmcj"
I20250628 20:03:37.795094 7841 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:37.824558 7841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:37.825259 7841 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:37.825433 7841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:37.826057 7841 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:37.827028 7841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:37.827075 7841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.827121 7841 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:37.827148 7841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:37.835942 7841 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:45903
I20250628 20:03:37.835969 7968 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:45903 every 8 connection(s)
I20250628 20:03:37.836944 7841 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 20:03:37.838860 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 7841
I20250628 20:03:37.838961 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 20:03:37.845953 7969 heartbeater.cc:344] Connected to a master server at 127.5.100.126:45935
I20250628 20:03:37.846185 7969 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:37.846696 7969 heartbeater.cc:507] Master 127.5.100.126:45935 requested a full tablet report, sending...
I20250628 20:03:37.847662 7519 ts_manager.cc:194] Registered new tserver with Master: 936c980484cb48568ec596a4127803e4 (127.5.100.67:45903)
I20250628 20:03:37.848193 7519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:51779
I20250628 20:03:37.853158 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:37.861445 7519 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34148:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 20:03:37.863937 7519 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 20:03:37.879699 7903 tablet_service.cc:1468] Processing CreateTablet for tablet b4b371a15ede4353a987b538b18081ea (DEFAULT_TABLE table=TestTable [id=89b24a015e3e4d28ad596758d635584c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:37.879699 7772 tablet_service.cc:1468] Processing CreateTablet for tablet b4b371a15ede4353a987b538b18081ea (DEFAULT_TABLE table=TestTable [id=89b24a015e3e4d28ad596758d635584c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:37.880036 7641 tablet_service.cc:1468] Processing CreateTablet for tablet b4b371a15ede4353a987b538b18081ea (DEFAULT_TABLE table=TestTable [id=89b24a015e3e4d28ad596758d635584c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:37.880651 7903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b371a15ede4353a987b538b18081ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:37.880658 7772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b371a15ede4353a987b538b18081ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:37.880754 7641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4b371a15ede4353a987b538b18081ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:37.885608 7986 tablet_bootstrap.cc:492] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4: Bootstrap starting.
I20250628 20:03:37.886839 7988 tablet_bootstrap.cc:492] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0: Bootstrap starting.
I20250628 20:03:37.888242 7986 tablet_bootstrap.cc:654] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:37.888316 7987 tablet_bootstrap.cc:492] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2: Bootstrap starting.
I20250628 20:03:37.888893 7988 tablet_bootstrap.cc:654] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:37.889197 7986 log.cc:826] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:37.889724 7988 log.cc:826] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:37.890206 7987 tablet_bootstrap.cc:654] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:37.890939 7986 tablet_bootstrap.cc:492] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4: No bootstrap required, opened a new log
I20250628 20:03:37.890950 7987 log.cc:826] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:37.891119 7986 ts_tablet_manager.cc:1397] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250628 20:03:37.891377 7988 tablet_bootstrap.cc:492] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0: No bootstrap required, opened a new log
I20250628 20:03:37.891551 7988 ts_tablet_manager.cc:1397] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250628 20:03:37.892637 7987 tablet_bootstrap.cc:492] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2: No bootstrap required, opened a new log
I20250628 20:03:37.892813 7987 ts_tablet_manager.cc:1397] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250628 20:03:37.893579 7986 raft_consensus.cc:357] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.893954 7986 raft_consensus.cc:383] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:37.894057 7986 raft_consensus.cc:738] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936c980484cb48568ec596a4127803e4, State: Initialized, Role: FOLLOWER
I20250628 20:03:37.894591 7986 consensus_queue.cc:260] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.894537 7988 raft_consensus.cc:357] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.894730 7988 raft_consensus.cc:383] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:37.894810 7988 raft_consensus.cc:738] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32650294f7f24aa0ad7cb240326417a0, State: Initialized, Role: FOLLOWER
I20250628 20:03:37.895292 7969 heartbeater.cc:499] Master 127.5.100.126:45935 was elected leader, sending a full tablet report...
I20250628 20:03:37.895421 7986 ts_tablet_manager.cc:1428] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250628 20:03:37.895644 7987 raft_consensus.cc:357] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.895839 7987 raft_consensus.cc:383] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:37.895903 7987 raft_consensus.cc:738] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527f6b9bdbf544c2935853102b9488c2, State: Initialized, Role: FOLLOWER
I20250628 20:03:37.896297 7987 consensus_queue.cc:260] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.896344 7988 consensus_queue.cc:260] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.897187 7987 ts_tablet_manager.cc:1428] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250628 20:03:37.897187 7988 ts_tablet_manager.cc:1428] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250628 20:03:37.897221 7838 heartbeater.cc:499] Master 127.5.100.126:45935 was elected leader, sending a full tablet report...
I20250628 20:03:37.897982 7707 heartbeater.cc:499] Master 127.5.100.126:45935 was elected leader, sending a full tablet report...
W20250628 20:03:37.904356 7839 tablet.cc:2378] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:37.964597 7993 raft_consensus.cc:491] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:37.964815 7993 raft_consensus.cc:513] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.965960 7993 leader_election.cc:290] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32650294f7f24aa0ad7cb240326417a0 (127.5.100.66:33477), 936c980484cb48568ec596a4127803e4 (127.5.100.67:45903)
I20250628 20:03:37.967747 7994 raft_consensus.cc:491] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:37.967957 7994 raft_consensus.cc:513] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.969198 7994 leader_election.cc:290] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 936c980484cb48568ec596a4127803e4 (127.5.100.67:45903), 527f6b9bdbf544c2935853102b9488c2 (127.5.100.65:33837)
I20250628 20:03:37.970369 7792 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b371a15ede4353a987b538b18081ea" candidate_uuid: "527f6b9bdbf544c2935853102b9488c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32650294f7f24aa0ad7cb240326417a0" is_pre_election: true
I20250628 20:03:37.970566 7792 raft_consensus.cc:2466] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527f6b9bdbf544c2935853102b9488c2 in term 0.
I20250628 20:03:37.971153 7596 leader_election.cc:304] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32650294f7f24aa0ad7cb240326417a0, 527f6b9bdbf544c2935853102b9488c2; no voters:
I20250628 20:03:37.971393 7923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b371a15ede4353a987b538b18081ea" candidate_uuid: "527f6b9bdbf544c2935853102b9488c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936c980484cb48568ec596a4127803e4" is_pre_election: true
I20250628 20:03:37.971514 7993 raft_consensus.cc:2802] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 20:03:37.971578 7993 raft_consensus.cc:491] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 20:03:37.971616 7993 raft_consensus.cc:3058] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:37.971668 7923 raft_consensus.cc:2466] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 527f6b9bdbf544c2935853102b9488c2 in term 0.
I20250628 20:03:37.972741 7993 raft_consensus.cc:513] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.973135 7993 leader_election.cc:290] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [CANDIDATE]: Term 1 election: Requested vote from peers 32650294f7f24aa0ad7cb240326417a0 (127.5.100.66:33477), 936c980484cb48568ec596a4127803e4 (127.5.100.67:45903)
I20250628 20:03:37.973315 7661 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b371a15ede4353a987b538b18081ea" candidate_uuid: "32650294f7f24aa0ad7cb240326417a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "527f6b9bdbf544c2935853102b9488c2" is_pre_election: true
I20250628 20:03:37.973448 7792 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b371a15ede4353a987b538b18081ea" candidate_uuid: "527f6b9bdbf544c2935853102b9488c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32650294f7f24aa0ad7cb240326417a0"
I20250628 20:03:37.973531 7661 raft_consensus.cc:2391] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32650294f7f24aa0ad7cb240326417a0 in current term 1: Already voted for candidate 527f6b9bdbf544c2935853102b9488c2 in this term.
I20250628 20:03:37.973558 7792 raft_consensus.cc:3058] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:37.973716 7923 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b371a15ede4353a987b538b18081ea" candidate_uuid: "527f6b9bdbf544c2935853102b9488c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936c980484cb48568ec596a4127803e4"
I20250628 20:03:37.973832 7923 raft_consensus.cc:3058] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:37.974622 7922 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b4b371a15ede4353a987b538b18081ea" candidate_uuid: "32650294f7f24aa0ad7cb240326417a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936c980484cb48568ec596a4127803e4" is_pre_election: true
I20250628 20:03:37.974673 7792 raft_consensus.cc:2466] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527f6b9bdbf544c2935853102b9488c2 in term 1.
I20250628 20:03:37.974818 7923 raft_consensus.cc:2466] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527f6b9bdbf544c2935853102b9488c2 in term 1.
I20250628 20:03:37.975018 7596 leader_election.cc:304] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32650294f7f24aa0ad7cb240326417a0, 527f6b9bdbf544c2935853102b9488c2; no voters:
I20250628 20:03:37.975167 7725 leader_election.cc:304] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32650294f7f24aa0ad7cb240326417a0; no voters: 527f6b9bdbf544c2935853102b9488c2, 936c980484cb48568ec596a4127803e4
I20250628 20:03:37.975232 7993 raft_consensus.cc:2802] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:37.975376 7994 raft_consensus.cc:2747] T b4b371a15ede4353a987b538b18081ea P 32650294f7f24aa0ad7cb240326417a0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250628 20:03:37.975421 7993 raft_consensus.cc:695] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [term 1 LEADER]: Becoming Leader. State: Replica: 527f6b9bdbf544c2935853102b9488c2, State: Running, Role: LEADER
I20250628 20:03:37.975713 7993 consensus_queue.cc:237] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } }
I20250628 20:03:37.978781 7519 catalog_manager.cc:5582] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 527f6b9bdbf544c2935853102b9488c2 (127.5.100.65). New cstate: current_term: 1 leader_uuid: "527f6b9bdbf544c2935853102b9488c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32650294f7f24aa0ad7cb240326417a0" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 33477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "936c980484cb48568ec596a4127803e4" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 45903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "527f6b9bdbf544c2935853102b9488c2" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 33837 } health_report { overall_health: HEALTHY } } }
W20250628 20:03:37.983184 7708 tablet.cc:2378] T b4b371a15ede4353a987b538b18081ea P 527f6b9bdbf544c2935853102b9488c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:37.997452 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:37.998811 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 527f6b9bdbf544c2935853102b9488c2 to finish bootstrapping
I20250628 20:03:38.004079 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 32650294f7f24aa0ad7cb240326417a0 to finish bootstrapping
I20250628 20:03:38.008646 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 936c980484cb48568ec596a4127803e4 to finish bootstrapping
W20250628 20:03:38.087882 7970 tablet.cc:2378] T b4b371a15ede4353a987b538b18081ea P 936c980484cb48568ec596a4127803e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 20:03:38.141132 7519 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:34168
I20250628 20:03:38.148080 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 7579
I20250628 20:03:38.153658 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 7710
I20250628 20:03:38.158856 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 7841
I20250628 20:03:38.163763 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 7488
2025-06-28T20:03:38Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (1118 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250628 20:03:38.183960 5521 test_util.cc:276] Using random seed: 1247226130
I20250628 20:03:38.184698 5521 ts_itest-base.cc:115] Starting cluster with:
I20250628 20:03:38.184744 5521 ts_itest-base.cc:116] --------------
I20250628 20:03:38.184754 5521 ts_itest-base.cc:117] 3 tablet servers
I20250628 20:03:38.184763 5521 ts_itest-base.cc:118] 3 replicas per TS
I20250628 20:03:38.184772 5521 ts_itest-base.cc:119] --------------
2025-06-28T20:03:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T20:03:38Z Disabled control of system clock
I20250628 20:03:38.192174 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:32981
--webserver_interface=127.5.100.126
--webserver_port=0
--builtin_ntp_servers=127.5.100.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:32981 with env {}
W20250628 20:03:38.297449 8023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:38.297739 8023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:38.297811 8023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:38.300963 8023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:38.301028 8023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:38.301054 8023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:38.301077 8023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:38.304836 8023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:32981
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:38.305953 8023 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:38.307085 8023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:38.313931 8028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:38.313964 8029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:38.314064 8031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:38.314121 8023 server_base.cc:1048] running on GCE node
I20250628 20:03:38.314626 8023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:38.315574 8023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:38.316726 8023 hybrid_clock.cc:648] HybridClock initialized: now 1751141018316715 us; error 30 us; skew 500 ppm
I20250628 20:03:38.318662 8023 webserver.cc:469] Webserver started at http://127.5.100.126:38357/ using document root <none> and password file <none>
I20250628 20:03:38.319231 8023 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:38.319288 8023 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:38.319471 8023 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:38.321060 8023 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9259eda8a8d3477d870ae81a6c681ad6"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.321542 8023 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9259eda8a8d3477d870ae81a6c681ad6"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.324941 8023 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20250628 20:03:38.327162 8037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.328184 8023 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250628 20:03:38.328292 8023 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9259eda8a8d3477d870ae81a6c681ad6"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.328378 8023 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:38.356179 8023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:38.356938 8023 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:38.357120 8023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:38.364369 8023 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:32981
I20250628 20:03:38.364384 8089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:32981 every 8 connection(s)
I20250628 20:03:38.365479 8023 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:38.367074 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 8023
I20250628 20:03:38.367178 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 20:03:38.368248 8090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:38.374879 8090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Bootstrap starting.
I20250628 20:03:38.377349 8090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:38.378064 8090 log.cc:826] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:38.379673 8090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: No bootstrap required, opened a new log
I20250628 20:03:38.382186 8090 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:38.382369 8090 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:38.382409 8090 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9259eda8a8d3477d870ae81a6c681ad6, State: Initialized, Role: FOLLOWER
I20250628 20:03:38.382757 8090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:38.382858 8090 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:38.382898 8090 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:38.382983 8090 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:38.383757 8090 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:38.384063 8090 leader_election.cc:304] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9259eda8a8d3477d870ae81a6c681ad6; no voters:
I20250628 20:03:38.384299 8090 leader_election.cc:290] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:38.384400 8095 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:38.384575 8095 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 1 LEADER]: Becoming Leader. State: Replica: 9259eda8a8d3477d870ae81a6c681ad6, State: Running, Role: LEADER
I20250628 20:03:38.384886 8095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:38.385386 8090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:38.386409 8096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9259eda8a8d3477d870ae81a6c681ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } } }
I20250628 20:03:38.386389 8097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9259eda8a8d3477d870ae81a6c681ad6. Latest consensus state: current_term: 1 leader_uuid: "9259eda8a8d3477d870ae81a6c681ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } } }
I20250628 20:03:38.386529 8096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:38.386533 8097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:38.386870 8104 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:38.389717 8104 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:38.394737 8104 catalog_manager.cc:1349] Generated new cluster ID: 35fb7e7b30c942348197e2e73158589f
I20250628 20:03:38.394812 8104 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:38.413021 8104 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:38.414058 8104 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:38.423434 8104 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Generated new TSK 0
I20250628 20:03:38.424199 8104 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:38.426901 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:0
--local_ip_for_outbound_sockets=127.5.100.65
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32981
--builtin_ntp_servers=127.5.100.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 20:03:38.529801 8114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:38.530047 8114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:38.530109 8114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:38.533396 8114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:38.533535 8114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:38.537387 8114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32981
--never_fsync=true
--heap_profile_path=/tmp/kudu.8114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:38.538540 8114 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:38.539690 8114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:38.547434 8122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:38.547518 8120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:38.547513 8119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:38.547727 8114 server_base.cc:1048] running on GCE node
I20250628 20:03:38.548143 8114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:38.548735 8114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:38.549876 8114 hybrid_clock.cc:648] HybridClock initialized: now 1751141018549859 us; error 30 us; skew 500 ppm
I20250628 20:03:38.551741 8114 webserver.cc:469] Webserver started at http://127.5.100.65:33689/ using document root <none> and password file <none>
I20250628 20:03:38.552314 8114 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:38.552372 8114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:38.552549 8114 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:38.554198 8114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cf4a534050334cb4a5061612ea4bb09a"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.554661 8114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cf4a534050334cb4a5061612ea4bb09a"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.558069 8114 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:38.560314 8128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.561362 8114 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:38.561480 8114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf4a534050334cb4a5061612ea4bb09a"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.561578 8114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:38.577510 8114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:38.578260 8114 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:38.578433 8114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:38.579000 8114 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:38.579949 8114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:38.579995 8114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.580050 8114 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:38.580073 8114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.588784 8114 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:46455
I20250628 20:03:38.588816 8241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:46455 every 8 connection(s)
I20250628 20:03:38.589776 8114 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 20:03:38.592682 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 8114
I20250628 20:03:38.592785 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 20:03:38.594836 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:0
--local_ip_for_outbound_sockets=127.5.100.66
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32981
--builtin_ntp_servers=127.5.100.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:38.599284 8242 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32981
I20250628 20:03:38.599596 8242 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:38.600363 8242 heartbeater.cc:507] Master 127.5.100.126:32981 requested a full tablet report, sending...
I20250628 20:03:38.602015 8054 ts_manager.cc:194] Registered new tserver with Master: cf4a534050334cb4a5061612ea4bb09a (127.5.100.65:46455)
I20250628 20:03:38.603186 8054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:58433
W20250628 20:03:38.700565 8245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:38.700799 8245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:38.700860 8245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:38.703996 8245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:38.704138 8245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:38.707834 8245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32981
--never_fsync=true
--heap_profile_path=/tmp/kudu.8245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:38.708930 8245 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:38.710103 8245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:38.717672 8253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:38.717852 8245 server_base.cc:1048] running on GCE node
W20250628 20:03:38.717795 8250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:38.717983 8251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:38.718402 8245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:38.718974 8245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:38.720116 8245 hybrid_clock.cc:648] HybridClock initialized: now 1751141018720064 us; error 72 us; skew 500 ppm
I20250628 20:03:38.721932 8245 webserver.cc:469] Webserver started at http://127.5.100.66:36521/ using document root <none> and password file <none>
I20250628 20:03:38.722487 8245 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:38.722546 8245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:38.722730 8245 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:38.724303 8245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f6e2ff61a3f2472181a9fbea5807b71a"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.724737 8245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f6e2ff61a3f2472181a9fbea5807b71a"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.727990 8245 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20250628 20:03:38.730227 8259 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.731237 8245 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:38.731349 8245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f6e2ff61a3f2472181a9fbea5807b71a"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.731436 8245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:38.743456 8245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:38.744102 8245 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:38.744278 8245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:38.744841 8245 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:38.745894 8245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:38.745940 8245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.745978 8245 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:38.745995 8245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.754384 8245 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:36237
I20250628 20:03:38.754415 8372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:36237 every 8 connection(s)
I20250628 20:03:38.755326 8245 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 20:03:38.759924 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 8245
I20250628 20:03:38.760022 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 20:03:38.762217 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:0
--local_ip_for_outbound_sockets=127.5.100.67
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32981
--builtin_ntp_servers=127.5.100.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:38.764547 8373 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32981
I20250628 20:03:38.764792 8373 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:38.765353 8373 heartbeater.cc:507] Master 127.5.100.126:32981 requested a full tablet report, sending...
I20250628 20:03:38.766446 8054 ts_manager.cc:194] Registered new tserver with Master: f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66:36237)
I20250628 20:03:38.767221 8054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:44347
W20250628 20:03:38.869481 8376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:38.869751 8376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:38.869814 8376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:38.873126 8376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:38.873248 8376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:38.877224 8376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:32981
--never_fsync=true
--heap_profile_path=/tmp/kudu.8376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:38.878417 8376 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:38.879544 8376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:38.887290 8382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:38.887281 8381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:38.887282 8384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:38.887455 8376 server_base.cc:1048] running on GCE node
I20250628 20:03:38.887991 8376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:38.888562 8376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:38.889745 8376 hybrid_clock.cc:648] HybridClock initialized: now 1751141018889720 us; error 38 us; skew 500 ppm
I20250628 20:03:38.891618 8376 webserver.cc:469] Webserver started at http://127.5.100.67:41683/ using document root <none> and password file <none>
I20250628 20:03:38.892197 8376 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:38.892258 8376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:38.892442 8376 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:38.894156 8376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "af0bbe42590542c4ad39fad4e8540a30"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.894632 8376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "af0bbe42590542c4ad39fad4e8540a30"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.898037 8376 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:38.900233 8390 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.901222 8376 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:38.901325 8376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "af0bbe42590542c4ad39fad4e8540a30"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:38.901410 8376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:38.928300 8376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:38.928993 8376 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:38.929168 8376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:38.929811 8376 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:38.930757 8376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:38.930805 8376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.930867 8376 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:38.930891 8376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:38.939531 8376 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:44543
I20250628 20:03:38.939551 8503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:44543 every 8 connection(s)
I20250628 20:03:38.940557 8376 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 20:03:38.948132 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 8376
I20250628 20:03:38.948261 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 20:03:38.950253 8504 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32981
I20250628 20:03:38.950529 8504 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:38.951174 8504 heartbeater.cc:507] Master 127.5.100.126:32981 requested a full tablet report, sending...
I20250628 20:03:38.952147 8054 ts_manager.cc:194] Registered new tserver with Master: af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67:44543)
I20250628 20:03:38.952677 8054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:52173
I20250628 20:03:38.962699 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:38.971606 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:38.971683 5521 test_util.cc:276] Using random seed: 1248013860
I20250628 20:03:38.980865 8054 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48316:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 20:03:38.995711 8176 tablet_service.cc:1468] Processing CreateTablet for tablet 6aab4b81ed5f4ed783a4931736a84615 (DEFAULT_TABLE table=TestTable [id=49afa85fcbc949468dbb0660ebd1e100]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:38.996776 8176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aab4b81ed5f4ed783a4931736a84615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:39.001864 8522 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Bootstrap starting.
I20250628 20:03:39.003739 8522 tablet_bootstrap.cc:654] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:39.004416 8522 log.cc:826] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:39.006160 8522 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: No bootstrap required, opened a new log
I20250628 20:03:39.006335 8522 ts_tablet_manager.cc:1397] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250628 20:03:39.008535 8522 raft_consensus.cc:357] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } }
I20250628 20:03:39.008713 8522 raft_consensus.cc:383] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:39.008764 8522 raft_consensus.cc:738] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4a534050334cb4a5061612ea4bb09a, State: Initialized, Role: FOLLOWER
I20250628 20:03:39.009168 8522 consensus_queue.cc:260] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } }
I20250628 20:03:39.009287 8522 raft_consensus.cc:397] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:39.009346 8522 raft_consensus.cc:491] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:39.009452 8522 raft_consensus.cc:3058] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:39.010241 8522 raft_consensus.cc:513] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } }
I20250628 20:03:39.010548 8522 leader_election.cc:304] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf4a534050334cb4a5061612ea4bb09a; no voters:
I20250628 20:03:39.010867 8522 leader_election.cc:290] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:39.010973 8524 raft_consensus.cc:2802] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:39.011119 8524 raft_consensus.cc:695] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 1 LEADER]: Becoming Leader. State: Replica: cf4a534050334cb4a5061612ea4bb09a, State: Running, Role: LEADER
I20250628 20:03:39.011382 8524 consensus_queue.cc:237] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } }
I20250628 20:03:39.011695 8522 ts_tablet_manager.cc:1428] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250628 20:03:39.011713 8242 heartbeater.cc:499] Master 127.5.100.126:32981 was elected leader, sending a full tablet report...
I20250628 20:03:39.014722 8054 catalog_manager.cc:5582] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a reported cstate change: term changed from 0 to 1, leader changed from <none> to cf4a534050334cb4a5061612ea4bb09a (127.5.100.65). New cstate: current_term: 1 leader_uuid: "cf4a534050334cb4a5061612ea4bb09a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:39.062877 5521 test_util.cc:276] Using random seed: 1248105043
I20250628 20:03:39.070334 8052 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48318:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 20:03:39.081022 8307 tablet_service.cc:1468] Processing CreateTablet for tablet 48df44a6e83744a3a976e52cd05ea91e (DEFAULT_TABLE table=TestTable1 [id=cba3d1c32fd2430bb35e0983881e777c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:39.082093 8307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48df44a6e83744a3a976e52cd05ea91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:39.087406 8544 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Bootstrap starting.
I20250628 20:03:39.089493 8544 tablet_bootstrap.cc:654] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:39.090247 8544 log.cc:826] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:39.091950 8544 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: No bootstrap required, opened a new log
I20250628 20:03:39.092115 8544 ts_tablet_manager.cc:1397] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250628 20:03:39.094533 8544 raft_consensus.cc:357] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } }
I20250628 20:03:39.094715 8544 raft_consensus.cc:383] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:39.094749 8544 raft_consensus.cc:738] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e2ff61a3f2472181a9fbea5807b71a, State: Initialized, Role: FOLLOWER
I20250628 20:03:39.095124 8544 consensus_queue.cc:260] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } }
I20250628 20:03:39.095235 8544 raft_consensus.cc:397] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:39.095279 8544 raft_consensus.cc:491] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:39.095361 8544 raft_consensus.cc:3058] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:39.096099 8544 raft_consensus.cc:513] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } }
I20250628 20:03:39.096400 8544 leader_election.cc:304] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6e2ff61a3f2472181a9fbea5807b71a; no voters:
I20250628 20:03:39.096654 8544 leader_election.cc:290] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:39.096786 8546 raft_consensus.cc:2802] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:39.096983 8546 raft_consensus.cc:695] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 1 LEADER]: Becoming Leader. State: Replica: f6e2ff61a3f2472181a9fbea5807b71a, State: Running, Role: LEADER
I20250628 20:03:39.097255 8373 heartbeater.cc:499] Master 127.5.100.126:32981 was elected leader, sending a full tablet report...
I20250628 20:03:39.097281 8546 consensus_queue.cc:237] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } }
I20250628 20:03:39.097481 8544 ts_tablet_manager.cc:1428] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20250628 20:03:39.100335 8052 catalog_manager.cc:5582] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a reported cstate change: term changed from 0 to 1, leader changed from <none> to f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66). New cstate: current_term: 1 leader_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:39.152007 5521 test_util.cc:276] Using random seed: 1248194171
I20250628 20:03:39.159574 8050 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48330:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 20:03:39.170320 8438 tablet_service.cc:1468] Processing CreateTablet for tablet 28b34ae82b984d7fb3e30ce8c45b4c8a (DEFAULT_TABLE table=TestTable2 [id=e22bc7e1b81b4447923ac457bd880e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:39.171243 8438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28b34ae82b984d7fb3e30ce8c45b4c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:39.176198 8566 tablet_bootstrap.cc:492] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Bootstrap starting.
I20250628 20:03:39.178304 8566 tablet_bootstrap.cc:654] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:39.179003 8566 log.cc:826] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:39.180593 8566 tablet_bootstrap.cc:492] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: No bootstrap required, opened a new log
I20250628 20:03:39.180778 8566 ts_tablet_manager.cc:1397] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250628 20:03:39.183370 8566 raft_consensus.cc:357] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } }
I20250628 20:03:39.183563 8566 raft_consensus.cc:383] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:39.183614 8566 raft_consensus.cc:738] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0bbe42590542c4ad39fad4e8540a30, State: Initialized, Role: FOLLOWER
I20250628 20:03:39.184005 8566 consensus_queue.cc:260] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } }
I20250628 20:03:39.184127 8566 raft_consensus.cc:397] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:39.184190 8566 raft_consensus.cc:491] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:39.184295 8566 raft_consensus.cc:3058] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:39.185075 8566 raft_consensus.cc:513] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } }
I20250628 20:03:39.185432 8566 leader_election.cc:304] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0bbe42590542c4ad39fad4e8540a30; no voters:
I20250628 20:03:39.185745 8566 leader_election.cc:290] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:39.185828 8568 raft_consensus.cc:2802] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:39.186034 8568 raft_consensus.cc:695] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 1 LEADER]: Becoming Leader. State: Replica: af0bbe42590542c4ad39fad4e8540a30, State: Running, Role: LEADER
I20250628 20:03:39.186350 8568 consensus_queue.cc:237] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } }
I20250628 20:03:39.186465 8504 heartbeater.cc:499] Master 127.5.100.126:32981 was elected leader, sending a full tablet report...
I20250628 20:03:39.186671 8566 ts_tablet_manager.cc:1428] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250628 20:03:39.189086 8050 catalog_manager.cc:5582] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 reported cstate change: term changed from 0 to 1, leader changed from <none> to af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67). New cstate: current_term: 1 leader_uuid: "af0bbe42590542c4ad39fad4e8540a30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:39.232642 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 8023
I20250628 20:03:39.876259 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 8114
I20250628 20:03:39.881503 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 8245
I20250628 20:03:39.886772 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 8376
I20250628 20:03:39.891948 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:32981
--webserver_interface=127.5.100.126
--webserver_port=38357
--builtin_ntp_servers=127.5.100.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:32981 with env {}
W20250628 20:03:39.993515 8625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:39.993764 8625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:39.993822 8625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:39.997054 8625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:39.997109 8625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:39.997136 8625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:39.997157 8625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:40.000880 8625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:32981
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:32981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=38357
--never_fsync=true
--heap_profile_path=/tmp/kudu.8625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:40.002156 8625 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:40.003268 8625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:40.009886 8633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.009876 8630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.009857 8631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:40.009935 8625 server_base.cc:1048] running on GCE node
I20250628 20:03:40.010537 8625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:40.011406 8625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:40.012548 8625 hybrid_clock.cc:648] HybridClock initialized: now 1751141020012539 us; error 23 us; skew 500 ppm
I20250628 20:03:40.014312 8625 webserver.cc:469] Webserver started at http://127.5.100.126:38357/ using document root <none> and password file <none>
I20250628 20:03:40.014820 8625 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:40.014868 8625 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:40.018185 8625 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:40.019992 8639 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:40.020965 8625 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:40.021056 8625 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9259eda8a8d3477d870ae81a6c681ad6"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:40.021430 8625 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:40.033351 8625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:40.034039 8625 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:40.034186 8625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:40.041028 8625 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:32981
I20250628 20:03:40.041035 8691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:32981 every 8 connection(s)
I20250628 20:03:40.042210 8625 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:40.045940 8692 sys_catalog.cc:263] Verifying existing consensus state
I20250628 20:03:40.046525 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 8625
I20250628 20:03:40.046963 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:46455
--local_ip_for_outbound_sockets=127.5.100.65
--tserver_master_addrs=127.5.100.126:32981
--webserver_port=33689
--webserver_interface=127.5.100.65
--builtin_ntp_servers=127.5.100.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:40.047758 8692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Bootstrap starting.
I20250628 20:03:40.052682 8692 log.cc:826] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:40.065047 8692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:40.065500 8692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Bootstrap complete.
I20250628 20:03:40.068281 8692 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:40.068601 8692 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9259eda8a8d3477d870ae81a6c681ad6, State: Initialized, Role: FOLLOWER
I20250628 20:03:40.069069 8692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:40.069183 8692 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:40.069238 8692 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:40.069336 8692 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 2 FOLLOWER]: Advancing to term 3
I20250628 20:03:40.070952 8692 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:40.071265 8692 leader_election.cc:304] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9259eda8a8d3477d870ae81a6c681ad6; no voters:
I20250628 20:03:40.071525 8692 leader_election.cc:290] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250628 20:03:40.071662 8696 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 3 FOLLOWER]: Leader election won for term 3
I20250628 20:03:40.071846 8696 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 3 LEADER]: Becoming Leader. State: Replica: 9259eda8a8d3477d870ae81a6c681ad6, State: Running, Role: LEADER
I20250628 20:03:40.072208 8696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:40.072674 8692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:40.073114 8696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9259eda8a8d3477d870ae81a6c681ad6. Latest consensus state: current_term: 3 leader_uuid: "9259eda8a8d3477d870ae81a6c681ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } } }
I20250628 20:03:40.073315 8696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:40.073443 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9259eda8a8d3477d870ae81a6c681ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } } }
I20250628 20:03:40.073546 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:40.073776 8701 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:40.077505 8701 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=27dfb80a91ab474d8b98a5017cb8ea2d]
I20250628 20:03:40.077929 8701 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=e22bc7e1b81b4447923ac457bd880e1d]
I20250628 20:03:40.078172 8701 catalog_manager.cc:671] Loaded metadata for table TestTable [id=f3a23a89402f4e038749853342911a67]
I20250628 20:03:40.079448 8701 tablet_loader.cc:96] loaded metadata for tablet 28b34ae82b984d7fb3e30ce8c45b4c8a (table TestTable2 [id=e22bc7e1b81b4447923ac457bd880e1d])
I20250628 20:03:40.079691 8701 tablet_loader.cc:96] loaded metadata for tablet 48df44a6e83744a3a976e52cd05ea91e (table TestTable1 [id=27dfb80a91ab474d8b98a5017cb8ea2d])
I20250628 20:03:40.079864 8701 tablet_loader.cc:96] loaded metadata for tablet 6aab4b81ed5f4ed783a4931736a84615 (table TestTable [id=f3a23a89402f4e038749853342911a67])
I20250628 20:03:40.079972 8701 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:40.080716 8701 catalog_manager.cc:1261] Loaded cluster ID: 35fb7e7b30c942348197e2e73158589f
I20250628 20:03:40.080765 8701 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:40.082616 8701 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:40.083233 8701 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Loaded TSK: 0
I20250628 20:03:40.084141 8701 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 20:03:40.164203 8694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:40.164431 8694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:40.164491 8694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:40.168098 8694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:40.168215 8694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:40.172026 8694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:46455
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=33689
--tserver_master_addrs=127.5.100.126:32981
--never_fsync=true
--heap_profile_path=/tmp/kudu.8694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:40.173139 8694 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:40.174360 8694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:40.181916 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.181978 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.181931 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:40.182751 8694 server_base.cc:1048] running on GCE node
I20250628 20:03:40.183151 8694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:40.183801 8694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:40.184962 8694 hybrid_clock.cc:648] HybridClock initialized: now 1751141020184943 us; error 30 us; skew 500 ppm
I20250628 20:03:40.186990 8694 webserver.cc:469] Webserver started at http://127.5.100.65:33689/ using document root <none> and password file <none>
I20250628 20:03:40.187551 8694 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:40.187625 8694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:40.190898 8694 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20250628 20:03:40.192883 8727 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:40.193958 8694 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:40.194067 8694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cf4a534050334cb4a5061612ea4bb09a"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:40.194440 8694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:40.218950 8694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:40.219617 8694 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:40.219782 8694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:40.220335 8694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:40.221542 8734 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250628 20:03:40.224666 8694 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 20:03:40.224720 8694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250628 20:03:40.224771 8694 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 20:03:40.226337 8694 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 20:03:40.226390 8694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250628 20:03:40.226708 8734 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Bootstrap starting.
I20250628 20:03:40.238054 8694 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:46455
I20250628 20:03:40.238234 8841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:46455 every 8 connection(s)
I20250628 20:03:40.239544 8694 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 20:03:40.243182 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 8694
I20250628 20:03:40.243660 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:36237
--local_ip_for_outbound_sockets=127.5.100.66
--tserver_master_addrs=127.5.100.126:32981
--webserver_port=36521
--webserver_interface=127.5.100.66
--builtin_ntp_servers=127.5.100.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:40.245214 8734 log.cc:826] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:40.253629 8842 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32981
I20250628 20:03:40.253948 8842 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:40.254630 8842 heartbeater.cc:507] Master 127.5.100.126:32981 requested a full tablet report, sending...
I20250628 20:03:40.257403 8656 ts_manager.cc:194] Registered new tserver with Master: cf4a534050334cb4a5061612ea4bb09a (127.5.100.65:46455)
I20250628 20:03:40.260197 8656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:55923
I20250628 20:03:40.267705 8734 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:40.268092 8734 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Bootstrap complete.
I20250628 20:03:40.268958 8734 ts_tablet_manager.cc:1397] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Time spent bootstrapping tablet: real 0.043s user 0.038s sys 0.000s
I20250628 20:03:40.270978 8734 raft_consensus.cc:357] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } }
I20250628 20:03:40.271278 8734 raft_consensus.cc:738] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf4a534050334cb4a5061612ea4bb09a, State: Initialized, Role: FOLLOWER
I20250628 20:03:40.271708 8734 consensus_queue.cc:260] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } }
I20250628 20:03:40.271828 8734 raft_consensus.cc:397] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:40.271888 8734 raft_consensus.cc:491] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:40.271992 8734 raft_consensus.cc:3058] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:40.273528 8734 raft_consensus.cc:513] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } }
I20250628 20:03:40.273875 8734 leader_election.cc:304] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf4a534050334cb4a5061612ea4bb09a; no voters:
I20250628 20:03:40.274232 8734 leader_election.cc:290] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 20:03:40.274295 8848 raft_consensus.cc:2802] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 2 FOLLOWER]: Leader election won for term 2
I20250628 20:03:40.274489 8848 raft_consensus.cc:695] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 2 LEADER]: Becoming Leader. State: Replica: cf4a534050334cb4a5061612ea4bb09a, State: Running, Role: LEADER
I20250628 20:03:40.274768 8848 consensus_queue.cc:237] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } }
I20250628 20:03:40.275134 8842 heartbeater.cc:499] Master 127.5.100.126:32981 was elected leader, sending a full tablet report...
I20250628 20:03:40.275170 8734 ts_tablet_manager.cc:1428] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
I20250628 20:03:40.277091 8656 catalog_manager.cc:5582] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a reported cstate change: term changed from 0 to 2, leader changed from <none> to cf4a534050334cb4a5061612ea4bb09a (127.5.100.65), VOTER cf4a534050334cb4a5061612ea4bb09a (127.5.100.65) added. New cstate: current_term: 2 leader_uuid: "cf4a534050334cb4a5061612ea4bb09a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } health_report { overall_health: HEALTHY } } }
W20250628 20:03:40.283335 8656 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6aab4b81ed5f4ed783a4931736a84615 with cas_config_opid_index -1: no extra replica candidate found for tablet 6aab4b81ed5f4ed783a4931736a84615 (table TestTable [id=f3a23a89402f4e038749853342911a67]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250628 20:03:40.353770 8846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:40.353994 8846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:40.354041 8846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:40.357156 8846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:40.357273 8846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:40.360826 8846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:36237
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=36521
--tserver_master_addrs=127.5.100.126:32981
--never_fsync=true
--heap_profile_path=/tmp/kudu.8846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:40.361845 8846 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:40.362946 8846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:40.370221 8859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.370235 8858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.370221 8861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:40.370566 8846 server_base.cc:1048] running on GCE node
I20250628 20:03:40.370911 8846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:40.371443 8846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:40.372584 8846 hybrid_clock.cc:648] HybridClock initialized: now 1751141020372571 us; error 25 us; skew 500 ppm
I20250628 20:03:40.374363 8846 webserver.cc:469] Webserver started at http://127.5.100.66:36521/ using document root <none> and password file <none>
I20250628 20:03:40.374917 8846 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:40.374987 8846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:40.378369 8846 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:40.380281 8867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:40.381223 8846 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:40.381323 8846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f6e2ff61a3f2472181a9fbea5807b71a"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:40.381714 8846 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:40.394515 8846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:40.395093 8846 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:40.395244 8846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:40.395717 8846 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:40.396879 8874 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250628 20:03:40.399942 8846 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 20:03:40.399996 8846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250628 20:03:40.400054 8846 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 20:03:40.401481 8846 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 20:03:40.401521 8846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250628 20:03:40.401834 8874 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Bootstrap starting.
I20250628 20:03:40.410871 8846 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:36237
I20250628 20:03:40.410912 8981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:36237 every 8 connection(s)
I20250628 20:03:40.411832 8846 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 20:03:40.412456 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 8846
I20250628 20:03:40.412811 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:44543
--local_ip_for_outbound_sockets=127.5.100.67
--tserver_master_addrs=127.5.100.126:32981
--webserver_port=41683
--webserver_interface=127.5.100.67
--builtin_ntp_servers=127.5.100.84:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 20:03:40.417302 8874 log.cc:826] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:40.421842 8982 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32981
I20250628 20:03:40.422190 8982 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:40.422735 8982 heartbeater.cc:507] Master 127.5.100.126:32981 requested a full tablet report, sending...
I20250628 20:03:40.424325 8656 ts_manager.cc:194] Registered new tserver with Master: f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66:36237)
I20250628 20:03:40.425056 8656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:41517
I20250628 20:03:40.442835 8874 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:40.443306 8874 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Bootstrap complete.
I20250628 20:03:40.444455 8874 ts_tablet_manager.cc:1397] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Time spent bootstrapping tablet: real 0.043s user 0.036s sys 0.004s
I20250628 20:03:40.447073 8874 raft_consensus.cc:357] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } }
I20250628 20:03:40.447466 8874 raft_consensus.cc:738] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e2ff61a3f2472181a9fbea5807b71a, State: Initialized, Role: FOLLOWER
I20250628 20:03:40.447958 8874 consensus_queue.cc:260] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } }
I20250628 20:03:40.448091 8874 raft_consensus.cc:397] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:40.448156 8874 raft_consensus.cc:491] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:40.448261 8874 raft_consensus.cc:3058] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:40.450259 8874 raft_consensus.cc:513] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } }
I20250628 20:03:40.450623 8874 leader_election.cc:304] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6e2ff61a3f2472181a9fbea5807b71a; no voters:
I20250628 20:03:40.450951 8874 leader_election.cc:290] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 20:03:40.451109 8987 raft_consensus.cc:2802] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 2 FOLLOWER]: Leader election won for term 2
I20250628 20:03:40.451308 8987 raft_consensus.cc:695] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 2 LEADER]: Becoming Leader. State: Replica: f6e2ff61a3f2472181a9fbea5807b71a, State: Running, Role: LEADER
I20250628 20:03:40.451623 8987 consensus_queue.cc:237] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } }
I20250628 20:03:40.451685 8982 heartbeater.cc:499] Master 127.5.100.126:32981 was elected leader, sending a full tablet report...
I20250628 20:03:40.451778 8874 ts_tablet_manager.cc:1428] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20250628 20:03:40.453930 8656 catalog_manager.cc:5582] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a reported cstate change: term changed from 0 to 2, leader changed from <none> to f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66), VOTER f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66) added. New cstate: current_term: 2 leader_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:40.462951 8936 consensus_queue.cc:237] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } }
I20250628 20:03:40.464087 8989 raft_consensus.cc:2953] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index -1 to 13, NON_VOTER cf4a534050334cb4a5061612ea4bb09a (127.5.100.65) added. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } } }
I20250628 20:03:40.466485 8643 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 48df44a6e83744a3a976e52cd05ea91e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250628 20:03:40.467300 8656 catalog_manager.cc:5582] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a reported cstate change: config changed from index -1 to 13, NON_VOTER cf4a534050334cb4a5061612ea4bb09a (127.5.100.65) added. New cstate: current_term: 2 leader_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250628 20:03:40.471182 8871 consensus_peers.cc:489] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a -> Peer cf4a534050334cb4a5061612ea4bb09a (127.5.100.65:46455): Couldn't send request to peer cf4a534050334cb4a5061612ea4bb09a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48df44a6e83744a3a976e52cd05ea91e. This is attempt 1: this message will repeat every 5th retry.
I20250628 20:03:40.495265 8796 consensus_queue.cc:237] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } }
I20250628 20:03:40.496534 8848 raft_consensus.cc:2953] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index -1 to 13, NON_VOTER f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66) added. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } } }
I20250628 20:03:40.498286 8643 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6aab4b81ed5f4ed783a4931736a84615 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250628 20:03:40.499245 8656 catalog_manager.cc:5582] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a reported cstate change: config changed from index -1 to 13, NON_VOTER f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66) added. New cstate: current_term: 2 leader_uuid: "cf4a534050334cb4a5061612ea4bb09a" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250628 20:03:40.501112 8731 consensus_peers.cc:489] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a -> Peer f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66:36237): Couldn't send request to peer f6e2ff61a3f2472181a9fbea5807b71a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6aab4b81ed5f4ed783a4931736a84615. This is attempt 1: this message will repeat every 5th retry.
W20250628 20:03:40.535413 8984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:40.535641 8984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:40.535698 8984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:40.538868 8984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:40.538997 8984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:40.542997 8984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:44543
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=41683
--tserver_master_addrs=127.5.100.126:32981
--never_fsync=true
--heap_profile_path=/tmp/kudu.8984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:40.544060 8984 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:40.545148 8984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:40.552476 8999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.552563 9002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.552481 9000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:40.552829 8984 server_base.cc:1048] running on GCE node
I20250628 20:03:40.553165 8984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:40.553717 8984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:40.554858 8984 hybrid_clock.cc:648] HybridClock initialized: now 1751141020554850 us; error 43 us; skew 500 ppm
I20250628 20:03:40.556527 8984 webserver.cc:469] Webserver started at http://127.5.100.67:41683/ using document root <none> and password file <none>
I20250628 20:03:40.557054 8984 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:40.557121 8984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:40.560380 8984 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:40.562259 9008 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:40.563200 8984 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:40.563297 8984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "af0bbe42590542c4ad39fad4e8540a30"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:40.563661 8984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:40.573227 8984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:40.573851 8984 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:40.574009 8984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:40.574551 8984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:40.575672 9015 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250628 20:03:40.578716 8984 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 20:03:40.578773 8984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250628 20:03:40.578845 8984 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 20:03:40.580260 8984 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 20:03:40.580302 8984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250628 20:03:40.580608 9015 tablet_bootstrap.cc:492] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Bootstrap starting.
I20250628 20:03:40.590046 8984 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:44543
I20250628 20:03:40.590073 9122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:44543 every 8 connection(s)
I20250628 20:03:40.591027 8984 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 20:03:40.591425 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 8984
I20250628 20:03:40.596558 9015 log.cc:826] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:40.600679 9123 heartbeater.cc:344] Connected to a master server at 127.5.100.126:32981
I20250628 20:03:40.600914 9123 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:40.601421 9123 heartbeater.cc:507] Master 127.5.100.126:32981 requested a full tablet report, sending...
I20250628 20:03:40.602957 8656 ts_manager.cc:194] Registered new tserver with Master: af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67:44543)
I20250628 20:03:40.603657 8656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:41367
I20250628 20:03:40.606221 8796 consensus_queue.cc:237] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:40.607124 8849 raft_consensus.cc:2953] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67) added. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } } }
I20250628 20:03:40.609230 8643 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6aab4b81ed5f4ed783a4931736a84615 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20250628 20:03:40.609534 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250628 20:03:40.609933 8731 consensus_peers.cc:489] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a -> Peer f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66:36237): Couldn't send request to peer f6e2ff61a3f2472181a9fbea5807b71a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6aab4b81ed5f4ed783a4931736a84615. This is attempt 1: this message will repeat every 5th retry.
I20250628 20:03:40.609911 8656 catalog_manager.cc:5582] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a reported cstate change: config changed from index 13 to 14, NON_VOTER af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67) added. New cstate: current_term: 2 leader_uuid: "cf4a534050334cb4a5061612ea4bb09a" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250628 20:03:40.611050 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:40.612632 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cf4a534050334cb4a5061612ea4bb09a to finish bootstrapping
I20250628 20:03:40.617832 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f6e2ff61a3f2472181a9fbea5807b71a to finish bootstrapping
I20250628 20:03:40.618378 9015 tablet_bootstrap.cc:492] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20250628 20:03:40.618469 8731 consensus_peers.cc:489] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a -> Peer af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67:44543): Couldn't send request to peer af0bbe42590542c4ad39fad4e8540a30. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6aab4b81ed5f4ed783a4931736a84615. This is attempt 1: this message will repeat every 5th retry.
I20250628 20:03:40.618762 9015 tablet_bootstrap.cc:492] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Bootstrap complete.
I20250628 20:03:40.619882 9015 ts_tablet_manager.cc:1397] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.005s
I20250628 20:03:40.622295 9015 raft_consensus.cc:357] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } }
I20250628 20:03:40.622685 9015 raft_consensus.cc:738] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0bbe42590542c4ad39fad4e8540a30, State: Initialized, Role: FOLLOWER
I20250628 20:03:40.623162 9015 consensus_queue.cc:260] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } }
I20250628 20:03:40.623306 9015 raft_consensus.cc:397] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:40.623379 9015 raft_consensus.cc:491] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:40.623485 9015 raft_consensus.cc:3058] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:40.623497 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver af0bbe42590542c4ad39fad4e8540a30 to finish bootstrapping
I20250628 20:03:40.625103 9015 raft_consensus.cc:513] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } }
I20250628 20:03:40.625428 9015 leader_election.cc:304] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af0bbe42590542c4ad39fad4e8540a30; no voters:
I20250628 20:03:40.625734 9015 leader_election.cc:290] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 20:03:40.625854 9130 raft_consensus.cc:2802] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 20:03:40.626017 9130 raft_consensus.cc:695] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [term 2 LEADER]: Becoming Leader. State: Replica: af0bbe42590542c4ad39fad4e8540a30, State: Running, Role: LEADER
I20250628 20:03:40.626303 9130 consensus_queue.cc:237] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } }
I20250628 20:03:40.626648 9015 ts_tablet_manager.cc:1428] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20250628 20:03:40.627053 9123 heartbeater.cc:499] Master 127.5.100.126:32981 was elected leader, sending a full tablet report...
I20250628 20:03:40.628489 8655 catalog_manager.cc:5582] T 28b34ae82b984d7fb3e30ce8c45b4c8a P af0bbe42590542c4ad39fad4e8540a30 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "af0bbe42590542c4ad39fad4e8540a30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 44543 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:40.667282 8936 consensus_queue.cc:237] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:40.668567 8989 raft_consensus.cc:2953] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67) added. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } } }
W20250628 20:03:40.669389 8871 consensus_peers.cc:489] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a -> Peer cf4a534050334cb4a5061612ea4bb09a (127.5.100.65:46455): Couldn't send request to peer cf4a534050334cb4a5061612ea4bb09a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48df44a6e83744a3a976e52cd05ea91e. This is attempt 1: this message will repeat every 5th retry.
I20250628 20:03:40.670044 8643 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 48df44a6e83744a3a976e52cd05ea91e with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250628 20:03:40.670876 8656 catalog_manager.cc:5582] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a reported cstate change: config changed from index 13 to 14, NON_VOTER af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67) added. New cstate: current_term: 2 leader_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250628 20:03:40.673497 8871 consensus_peers.cc:489] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a -> Peer af0bbe42590542c4ad39fad4e8540a30 (127.5.100.67:44543): Couldn't send request to peer af0bbe42590542c4ad39fad4e8540a30. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48df44a6e83744a3a976e52cd05ea91e. This is attempt 1: this message will repeat every 5th retry.
I20250628 20:03:40.686157 8774 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:40.686164 9057 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:40.686800 8916 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
9259eda8a8d3477d870ae81a6c681ad6 | 127.5.100.126:32981 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.100.84:43629 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
af0bbe42590542c4ad39fad4e8540a30 | 127.5.100.67:44543 | HEALTHY | <none> | 1 | 0
cf4a534050334cb4a5061612ea4bb09a | 127.5.100.65:46455 | HEALTHY | <none> | 1 | 0
f6e2ff61a3f2472181a9fbea5807b71a | 127.5.100.66:36237 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.100.65 | experimental | 127.5.100.65:46455
local_ip_for_outbound_sockets | 127.5.100.66 | experimental | 127.5.100.66:36237
local_ip_for_outbound_sockets | 127.5.100.67 | experimental | 127.5.100.67:44543
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.100.65:46455
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.100.66:36237
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.100.67:44543
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.100.84:43629 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 20:03:40.711354 5521 log_verifier.cc:126] Checking tablet 28b34ae82b984d7fb3e30ce8c45b4c8a
I20250628 20:03:40.722025 5521 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 28b34ae82b984d7fb3e30ce8c45b4c8a
I20250628 20:03:40.722121 5521 log_verifier.cc:126] Checking tablet 48df44a6e83744a3a976e52cd05ea91e
I20250628 20:03:40.730530 5521 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 48df44a6e83744a3a976e52cd05ea91e
I20250628 20:03:40.730592 5521 log_verifier.cc:126] Checking tablet 6aab4b81ed5f4ed783a4931736a84615
I20250628 20:03:40.738945 5521 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 6aab4b81ed5f4ed783a4931736a84615
I20250628 20:03:40.739097 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 8625
I20250628 20:03:40.745529 5521 minidump.cc:252] Setting minidump size limit to 20M
I20250628 20:03:40.746019 5521 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:40.746495 5521 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:40.751068 9161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:40.751111 9163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:40.751179 5521 server_base.cc:1048] running on GCE node
W20250628 20:03:40.751068 9160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:40.751750 5521 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250628 20:03:40.751859 5521 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250628 20:03:40.751899 5521 hybrid_clock.cc:648] HybridClock initialized: now 1751141020751897 us; error 0 us; skew 500 ppm
I20250628 20:03:40.753204 5521 webserver.cc:469] Webserver started at http://0.0.0.0:38063/ using document root <none> and password file <none>
I20250628 20:03:40.753831 5521 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:40.753881 5521 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:40.756438 5521 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20250628 20:03:40.758481 9168 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:40.759370 5521 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250628 20:03:40.759481 5521 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9259eda8a8d3477d870ae81a6c681ad6"
format_stamp: "Formatted at 2025-06-28 20:03:38 on dist-test-slave-rmcj"
I20250628 20:03:40.759729 5521 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:40.778208 5521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:40.778816 5521 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:40.778970 5521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:40.781786 5521 sys_catalog.cc:263] Verifying existing consensus state
W20250628 20:03:40.782524 5521 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.5.100.126:32981 exists but no master address supplied!
I20250628 20:03:40.783780 5521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Bootstrap starting.
I20250628 20:03:40.796140 5521 log.cc:826] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:40.811298 5521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=13 ignored=0} mutations{seen=17 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:40.811666 5521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6: Bootstrap complete.
I20250628 20:03:40.814643 5521 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:40.814810 5521 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9259eda8a8d3477d870ae81a6c681ad6, State: Initialized, Role: FOLLOWER
I20250628 20:03:40.815219 5521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:40.815325 5521 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:40.815382 5521 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:40.815485 5521 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 3 FOLLOWER]: Advancing to term 4
I20250628 20:03:40.816884 5521 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:40.817198 5521 leader_election.cc:304] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9259eda8a8d3477d870ae81a6c681ad6; no voters:
I20250628 20:03:40.817479 5521 leader_election.cc:290] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250628 20:03:40.817626 9175 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 4 FOLLOWER]: Leader election won for term 4
I20250628 20:03:40.817894 9175 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 4 LEADER]: Becoming Leader. State: Replica: 9259eda8a8d3477d870ae81a6c681ad6, State: Running, Role: LEADER
I20250628 20:03:40.818167 9175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 3.26, Last appended by leader: 26, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } }
I20250628 20:03:40.819473 9176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "9259eda8a8d3477d870ae81a6c681ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } } }
I20250628 20:03:40.819501 9177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9259eda8a8d3477d870ae81a6c681ad6. Latest consensus state: current_term: 4 leader_uuid: "9259eda8a8d3477d870ae81a6c681ad6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9259eda8a8d3477d870ae81a6c681ad6" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 32981 } } }
I20250628 20:03:40.819583 9176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:40.819597 9177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:40.826768 5521 tablet_replica.cc:331] stopping tablet replica
I20250628 20:03:40.827023 5521 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 4 LEADER]: Raft consensus shutting down.
I20250628 20:03:40.827176 5521 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 9259eda8a8d3477d870ae81a6c681ad6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250628 20:03:40.828224 5521 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250628 20:03:40.828480 5521 acceptor_pool.cc:194] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250628 20:03:40.841398 5521 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250628 20:03:40.996426 9211 ts_tablet_manager.cc:927] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: Initiating tablet copy from peer cf4a534050334cb4a5061612ea4bb09a (127.5.100.65:46455)
I20250628 20:03:40.997000 9211 tablet_copy_client.cc:323] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: tablet copy: Beginning tablet copy session from remote peer at address 127.5.100.65:46455
I20250628 20:03:41.000996 8816 tablet_copy_service.cc:140] P cf4a534050334cb4a5061612ea4bb09a: Received BeginTabletCopySession request for tablet 6aab4b81ed5f4ed783a4931736a84615 from peer af0bbe42590542c4ad39fad4e8540a30 ({username='slave'} at 127.5.100.67:55443)
I20250628 20:03:41.001156 8816 tablet_copy_service.cc:161] P cf4a534050334cb4a5061612ea4bb09a: Beginning new tablet copy session on tablet 6aab4b81ed5f4ed783a4931736a84615 from peer af0bbe42590542c4ad39fad4e8540a30 at {username='slave'} at 127.5.100.67:55443: session id = af0bbe42590542c4ad39fad4e8540a30-6aab4b81ed5f4ed783a4931736a84615
I20250628 20:03:41.002471 8816 tablet_copy_source_session.cc:215] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 20:03:41.004205 9211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aab4b81ed5f4ed783a4931736a84615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:41.007220 9211 tablet_copy_client.cc:806] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: tablet copy: Starting download of 0 data blocks...
I20250628 20:03:41.007483 9211 tablet_copy_client.cc:670] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: tablet copy: Starting download of 1 WAL segments...
I20250628 20:03:41.009254 9211 tablet_copy_client.cc:538] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 20:03:41.010663 9211 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: Bootstrap starting.
I20250628 20:03:41.020403 9219 ts_tablet_manager.cc:927] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: Initiating tablet copy from peer cf4a534050334cb4a5061612ea4bb09a (127.5.100.65:46455)
I20250628 20:03:41.020946 9219 tablet_copy_client.cc:323] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.100.65:46455
I20250628 20:03:41.021546 8816 tablet_copy_service.cc:140] P cf4a534050334cb4a5061612ea4bb09a: Received BeginTabletCopySession request for tablet 6aab4b81ed5f4ed783a4931736a84615 from peer f6e2ff61a3f2472181a9fbea5807b71a ({username='slave'} at 127.5.100.66:58513)
I20250628 20:03:41.021651 8816 tablet_copy_service.cc:161] P cf4a534050334cb4a5061612ea4bb09a: Beginning new tablet copy session on tablet 6aab4b81ed5f4ed783a4931736a84615 from peer f6e2ff61a3f2472181a9fbea5807b71a at {username='slave'} at 127.5.100.66:58513: session id = f6e2ff61a3f2472181a9fbea5807b71a-6aab4b81ed5f4ed783a4931736a84615
I20250628 20:03:41.022367 8816 tablet_copy_source_session.cc:215] T 6aab4b81ed5f4ed783a4931736a84615 P cf4a534050334cb4a5061612ea4bb09a: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 20:03:41.023679 9219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aab4b81ed5f4ed783a4931736a84615. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:41.026299 9219 tablet_copy_client.cc:806] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: tablet copy: Starting download of 0 data blocks...
I20250628 20:03:41.026530 9219 tablet_copy_client.cc:670] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: tablet copy: Starting download of 1 WAL segments...
I20250628 20:03:41.027886 9219 tablet_copy_client.cc:538] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 20:03:41.029001 9219 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: Bootstrap starting.
I20250628 20:03:41.032970 9211 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:41.033226 9211 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: Bootstrap complete.
I20250628 20:03:41.033370 9211 ts_tablet_manager.cc:1397] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.001s
I20250628 20:03:41.033941 9211 raft_consensus.cc:357] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:41.034061 9211 raft_consensus.cc:738] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: af0bbe42590542c4ad39fad4e8540a30, State: Initialized, Role: LEARNER
I20250628 20:03:41.034199 9211 consensus_queue.cc:260] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:41.034528 9211 ts_tablet_manager.cc:1428] T 6aab4b81ed5f4ed783a4931736a84615 P af0bbe42590542c4ad39fad4e8540a30: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250628 20:03:41.035300 8816 tablet_copy_service.cc:342] P cf4a534050334cb4a5061612ea4bb09a: Request end of tablet copy session af0bbe42590542c4ad39fad4e8540a30-6aab4b81ed5f4ed783a4931736a84615 received from {username='slave'} at 127.5.100.67:55443
I20250628 20:03:41.035418 8816 tablet_copy_service.cc:434] P cf4a534050334cb4a5061612ea4bb09a: ending tablet copy session af0bbe42590542c4ad39fad4e8540a30-6aab4b81ed5f4ed783a4931736a84615 on tablet 6aab4b81ed5f4ed783a4931736a84615 with peer af0bbe42590542c4ad39fad4e8540a30
W20250628 20:03:41.035509 9123 heartbeater.cc:646] Failed to heartbeat to 127.5.100.126:32981 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.100.126:32981: connect: Connection refused (error 111)
I20250628 20:03:41.046984 9219 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:41.047255 9219 tablet_bootstrap.cc:492] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: Bootstrap complete.
I20250628 20:03:41.047412 9219 ts_tablet_manager.cc:1397] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.004s
I20250628 20:03:41.047922 9219 raft_consensus.cc:357] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:41.048029 9219 raft_consensus.cc:738] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f6e2ff61a3f2472181a9fbea5807b71a, State: Initialized, Role: LEARNER
I20250628 20:03:41.048125 9219 consensus_queue.cc:260] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 46455 } } peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: NON_VOTER last_known_addr { host: "127.5.100.66" port: 36237 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:41.048367 9219 ts_tablet_manager.cc:1428] T 6aab4b81ed5f4ed783a4931736a84615 P f6e2ff61a3f2472181a9fbea5807b71a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250628 20:03:41.048974 8816 tablet_copy_service.cc:342] P cf4a534050334cb4a5061612ea4bb09a: Request end of tablet copy session f6e2ff61a3f2472181a9fbea5807b71a-6aab4b81ed5f4ed783a4931736a84615 received from {username='slave'} at 127.5.100.66:58513
I20250628 20:03:41.049064 8816 tablet_copy_service.cc:434] P cf4a534050334cb4a5061612ea4bb09a: ending tablet copy session f6e2ff61a3f2472181a9fbea5807b71a-6aab4b81ed5f4ed783a4931736a84615 on tablet 6aab4b81ed5f4ed783a4931736a84615 with peer f6e2ff61a3f2472181a9fbea5807b71a
W20250628 20:03:41.049247 8982 heartbeater.cc:646] Failed to heartbeat to 127.5.100.126:32981 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.100.126:32981: connect: Connection refused (error 111)
I20250628 20:03:41.189291 9211 ts_tablet_manager.cc:927] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: Initiating tablet copy from peer f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66:36237)
I20250628 20:03:41.189677 9211 tablet_copy_client.cc:323] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: tablet copy: Beginning tablet copy session from remote peer at address 127.5.100.66:36237
I20250628 20:03:41.193048 8956 tablet_copy_service.cc:140] P f6e2ff61a3f2472181a9fbea5807b71a: Received BeginTabletCopySession request for tablet 48df44a6e83744a3a976e52cd05ea91e from peer af0bbe42590542c4ad39fad4e8540a30 ({username='slave'} at 127.5.100.67:43031)
I20250628 20:03:41.193207 8956 tablet_copy_service.cc:161] P f6e2ff61a3f2472181a9fbea5807b71a: Beginning new tablet copy session on tablet 48df44a6e83744a3a976e52cd05ea91e from peer af0bbe42590542c4ad39fad4e8540a30 at {username='slave'} at 127.5.100.67:43031: session id = af0bbe42590542c4ad39fad4e8540a30-48df44a6e83744a3a976e52cd05ea91e
I20250628 20:03:41.194286 8956 tablet_copy_source_session.cc:215] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 20:03:41.195361 9211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48df44a6e83744a3a976e52cd05ea91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:41.197870 9211 tablet_copy_client.cc:806] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: tablet copy: Starting download of 0 data blocks...
I20250628 20:03:41.198112 9211 tablet_copy_client.cc:670] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: tablet copy: Starting download of 1 WAL segments...
I20250628 20:03:41.199671 9211 tablet_copy_client.cc:538] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 20:03:41.201077 9211 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: Bootstrap starting.
I20250628 20:03:41.232357 9211 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:41.232720 9211 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: Bootstrap complete.
I20250628 20:03:41.232935 9211 ts_tablet_manager.cc:1397] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.003s
I20250628 20:03:41.233522 9211 raft_consensus.cc:357] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:41.233675 9211 raft_consensus.cc:738] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: af0bbe42590542c4ad39fad4e8540a30, State: Initialized, Role: LEARNER
I20250628 20:03:41.233834 9211 consensus_queue.cc:260] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:41.234170 9211 ts_tablet_manager.cc:1428] T 48df44a6e83744a3a976e52cd05ea91e P af0bbe42590542c4ad39fad4e8540a30: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20250628 20:03:41.235095 8956 tablet_copy_service.cc:342] P f6e2ff61a3f2472181a9fbea5807b71a: Request end of tablet copy session af0bbe42590542c4ad39fad4e8540a30-48df44a6e83744a3a976e52cd05ea91e received from {username='slave'} at 127.5.100.67:43031
I20250628 20:03:41.235211 8956 tablet_copy_service.cc:434] P f6e2ff61a3f2472181a9fbea5807b71a: ending tablet copy session af0bbe42590542c4ad39fad4e8540a30-48df44a6e83744a3a976e52cd05ea91e on tablet 48df44a6e83744a3a976e52cd05ea91e with peer af0bbe42590542c4ad39fad4e8540a30
I20250628 20:03:41.266825 9233 ts_tablet_manager.cc:927] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: Initiating tablet copy from peer f6e2ff61a3f2472181a9fbea5807b71a (127.5.100.66:36237)
I20250628 20:03:41.267378 9233 tablet_copy_client.cc:323] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.100.66:36237
I20250628 20:03:41.267998 8956 tablet_copy_service.cc:140] P f6e2ff61a3f2472181a9fbea5807b71a: Received BeginTabletCopySession request for tablet 48df44a6e83744a3a976e52cd05ea91e from peer cf4a534050334cb4a5061612ea4bb09a ({username='slave'} at 127.5.100.65:43881)
I20250628 20:03:41.268103 8956 tablet_copy_service.cc:161] P f6e2ff61a3f2472181a9fbea5807b71a: Beginning new tablet copy session on tablet 48df44a6e83744a3a976e52cd05ea91e from peer cf4a534050334cb4a5061612ea4bb09a at {username='slave'} at 127.5.100.65:43881: session id = cf4a534050334cb4a5061612ea4bb09a-48df44a6e83744a3a976e52cd05ea91e
I20250628 20:03:41.268790 8956 tablet_copy_source_session.cc:215] T 48df44a6e83744a3a976e52cd05ea91e P f6e2ff61a3f2472181a9fbea5807b71a: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 20:03:41.270044 9233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48df44a6e83744a3a976e52cd05ea91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:41.272264 9233 tablet_copy_client.cc:806] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: tablet copy: Starting download of 0 data blocks...
I20250628 20:03:41.272434 9233 tablet_copy_client.cc:670] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: tablet copy: Starting download of 1 WAL segments...
I20250628 20:03:41.273979 9233 tablet_copy_client.cc:538] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 20:03:41.275162 9233 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: Bootstrap starting.
I20250628 20:03:41.293176 9233 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:41.293458 9233 tablet_bootstrap.cc:492] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: Bootstrap complete.
I20250628 20:03:41.293627 9233 ts_tablet_manager.cc:1397] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20250628 20:03:41.294210 9233 raft_consensus.cc:357] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:41.294317 9233 raft_consensus.cc:738] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf4a534050334cb4a5061612ea4bb09a, State: Initialized, Role: LEARNER
I20250628 20:03:41.294397 9233 consensus_queue.cc:260] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "f6e2ff61a3f2472181a9fbea5807b71a" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36237 } } peers { permanent_uuid: "cf4a534050334cb4a5061612ea4bb09a" member_type: NON_VOTER last_known_addr { host: "127.5.100.65" port: 46455 } attrs { promote: true } } peers { permanent_uuid: "af0bbe42590542c4ad39fad4e8540a30" member_type: NON_VOTER last_known_addr { host: "127.5.100.67" port: 44543 } attrs { promote: true } }
I20250628 20:03:41.294667 9233 ts_tablet_manager.cc:1428] T 48df44a6e83744a3a976e52cd05ea91e P cf4a534050334cb4a5061612ea4bb09a: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20250628 20:03:41.295382 8956 tablet_copy_service.cc:342] P f6e2ff61a3f2472181a9fbea5807b71a: Request end of tablet copy session cf4a534050334cb4a5061612ea4bb09a-48df44a6e83744a3a976e52cd05ea91e received from {username='slave'} at 127.5.100.65:43881
I20250628 20:03:41.295464 8956 tablet_copy_service.cc:434] P f6e2ff61a3f2472181a9fbea5807b71a: ending tablet copy session cf4a534050334cb4a5061612ea4bb09a-48df44a6e83744a3a976e52cd05ea91e on tablet 48df44a6e83744a3a976e52cd05ea91e with peer cf4a534050334cb4a5061612ea4bb09a
W20250628 20:03:41.295948 8842 heartbeater.cc:646] Failed to heartbeat to 127.5.100.126:32981 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.100.126:32981: connect: Connection refused (error 111)
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/kudu-admin-test.cc:3908: Failure
Value of: stdout2
Expected: has substring "Rebuilt from 3 replicas, of which 0 had errors"
Actual: "Rebuild Report\nTablet Servers\n address | status\n--------------------+--------\n 127.5.100.65:46455 | OK\n 127.5.100.66:36237 | OK\n 127.5.100.67:44543 | OK\nRebuilt from 3 tablet servers, of which 0 had errors\n\nReplicas\n table | tablet | tablet server | status\n-----------+----------------------------------+--------------------+--------\n TestTable | 6aab4b81ed5f4ed783a4931736a84615 | 127.5.100.65:46455 | OK\nRebuilt from 1 replicas, of which 0 had errors\n"
I20250628 20:03:41.332631 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 8694
I20250628 20:03:41.338357 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 8846
I20250628 20:03:41.343622 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 8984
2025-06-28T20:03:41Z chronyd exiting
I20250628 20:03:41.360198 5521 test_util.cc:183] -----------------------------------------------
I20250628 20:03:41.360257 5521 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751141006956821-5521-0
[ FAILED ] AdminCliTest.TestRebuildTables (3176 ms)
[----------] 5 tests from AdminCliTest (14393 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250628 20:03:41.360908 5521 test_util.cc:276] Using random seed: 1250403080
I20250628 20:03:41.361634 5521 ts_itest-base.cc:115] Starting cluster with:
I20250628 20:03:41.361666 5521 ts_itest-base.cc:116] --------------
I20250628 20:03:41.361677 5521 ts_itest-base.cc:117] 5 tablet servers
I20250628 20:03:41.361712 5521 ts_itest-base.cc:118] 3 replicas per TS
I20250628 20:03:41.361732 5521 ts_itest-base.cc:119] --------------
2025-06-28T20:03:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T20:03:41Z Disabled control of system clock
I20250628 20:03:41.370164 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:39089
--webserver_interface=127.5.100.126
--webserver_port=0
--builtin_ntp_servers=127.5.100.84:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:39089
--raft_prepare_replacement_before_eviction=true with env {}
W20250628 20:03:41.474458 9244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:41.474716 9244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:41.474774 9244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:41.478067 9244 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 20:03:41.478155 9244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:41.478184 9244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:41.478211 9244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:41.478245 9244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:41.482174 9244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:39089
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:41.483335 9244 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:41.484494 9244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:41.491219 9252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:41.491294 9249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:41.491441 9244 server_base.cc:1048] running on GCE node
W20250628 20:03:41.491225 9250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:41.492110 9244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:41.493021 9244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:41.494200 9244 hybrid_clock.cc:648] HybridClock initialized: now 1751141021494184 us; error 27 us; skew 500 ppm
I20250628 20:03:41.496287 9244 webserver.cc:469] Webserver started at http://127.5.100.126:46203/ using document root <none> and password file <none>
I20250628 20:03:41.496901 9244 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:41.496959 9244 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:41.497167 9244 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:41.498857 9244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "63be5c84fc5a467589793f0b497ba44d"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.499354 9244 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "63be5c84fc5a467589793f0b497ba44d"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.502795 9244 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250628 20:03:41.505149 9258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:41.506253 9244 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:41.506378 9244 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "63be5c84fc5a467589793f0b497ba44d"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.506470 9244 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:41.525022 9244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:41.525818 9244 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:41.526003 9244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:41.533200 9244 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:39089
I20250628 20:03:41.533202 9310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:39089 every 8 connection(s)
I20250628 20:03:41.534317 9244 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:41.535596 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 9244
I20250628 20:03:41.535701 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 20:03:41.537302 9311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:41.543273 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d: Bootstrap starting.
I20250628 20:03:41.545482 9311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:41.546252 9311 log.cc:826] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:41.547942 9311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d: No bootstrap required, opened a new log
I20250628 20:03:41.550522 9311 raft_consensus.cc:357] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63be5c84fc5a467589793f0b497ba44d" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 39089 } }
I20250628 20:03:41.550768 9311 raft_consensus.cc:383] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:41.550843 9311 raft_consensus.cc:738] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63be5c84fc5a467589793f0b497ba44d, State: Initialized, Role: FOLLOWER
I20250628 20:03:41.551278 9311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63be5c84fc5a467589793f0b497ba44d" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 39089 } }
I20250628 20:03:41.551406 9311 raft_consensus.cc:397] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:41.551488 9311 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:41.551595 9311 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:41.552397 9311 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63be5c84fc5a467589793f0b497ba44d" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 39089 } }
I20250628 20:03:41.552726 9311 leader_election.cc:304] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63be5c84fc5a467589793f0b497ba44d; no voters:
I20250628 20:03:41.553009 9311 leader_election.cc:290] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:41.553131 9316 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:41.553323 9316 raft_consensus.cc:695] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [term 1 LEADER]: Becoming Leader. State: Replica: 63be5c84fc5a467589793f0b497ba44d, State: Running, Role: LEADER
I20250628 20:03:41.553651 9316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63be5c84fc5a467589793f0b497ba44d" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 39089 } }
I20250628 20:03:41.554142 9311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:41.555313 9318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63be5c84fc5a467589793f0b497ba44d. Latest consensus state: current_term: 1 leader_uuid: "63be5c84fc5a467589793f0b497ba44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63be5c84fc5a467589793f0b497ba44d" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 39089 } } }
I20250628 20:03:41.555408 9317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63be5c84fc5a467589793f0b497ba44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63be5c84fc5a467589793f0b497ba44d" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 39089 } } }
I20250628 20:03:41.555532 9317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:41.555899 9325 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:41.556249 9318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:41.558786 9325 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:41.563745 9325 catalog_manager.cc:1349] Generated new cluster ID: 9edcf3b16daf4a2db0d8842457246d4a
I20250628 20:03:41.563848 9325 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:41.575372 9325 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:41.576318 9325 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:41.581809 9325 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 63be5c84fc5a467589793f0b497ba44d: Generated new TSK 0
I20250628 20:03:41.582559 9325 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:41.589120 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:0
--local_ip_for_outbound_sockets=127.5.100.65
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--builtin_ntp_servers=127.5.100.84:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250628 20:03:41.695835 9335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:41.696069 9335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:41.696121 9335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:41.699417 9335 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 20:03:41.699488 9335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:41.699570 9335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:41.703358 9335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--never_fsync=true
--heap_profile_path=/tmp/kudu.9335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:41.704595 9335 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:41.705799 9335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:41.713802 9340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:41.713785 9343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:41.713991 9335 server_base.cc:1048] running on GCE node
W20250628 20:03:41.714087 9341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:41.714473 9335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:41.715065 9335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:41.716218 9335 hybrid_clock.cc:648] HybridClock initialized: now 1751141021716205 us; error 26 us; skew 500 ppm
I20250628 20:03:41.718115 9335 webserver.cc:469] Webserver started at http://127.5.100.65:33759/ using document root <none> and password file <none>
I20250628 20:03:41.718683 9335 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:41.718740 9335 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:41.718922 9335 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:41.720512 9335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e761412b08874183a384aab630f61064"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.720968 9335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e761412b08874183a384aab630f61064"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.724268 9335 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250628 20:03:41.726469 9349 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:41.727452 9335 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:41.727555 9335 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e761412b08874183a384aab630f61064"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.727643 9335 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:41.743750 9335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:41.744447 9335 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:41.744616 9335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:41.745177 9335 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:41.746196 9335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:41.746244 9335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:41.746299 9335 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:41.746322 9335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:41.754774 9335 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:44187
I20250628 20:03:41.754817 9462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:44187 every 8 connection(s)
I20250628 20:03:41.755806 9335 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 20:03:41.764333 9463 heartbeater.cc:344] Connected to a master server at 127.5.100.126:39089
I20250628 20:03:41.764582 9463 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:41.765138 9463 heartbeater.cc:507] Master 127.5.100.126:39089 requested a full tablet report, sending...
I20250628 20:03:41.765410 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 9335
I20250628 20:03:41.765511 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 20:03:41.766782 9272 ts_manager.cc:194] Registered new tserver with Master: e761412b08874183a384aab630f61064 (127.5.100.65:44187)
I20250628 20:03:41.767762 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:0
--local_ip_for_outbound_sockets=127.5.100.66
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--builtin_ntp_servers=127.5.100.84:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 20:03:41.768062 9272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:34651
W20250628 20:03:41.871708 9466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:41.871944 9466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:41.871991 9466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:41.875162 9466 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 20:03:41.875237 9466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:41.875320 9466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:41.879067 9466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--never_fsync=true
--heap_profile_path=/tmp/kudu.9466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:41.880180 9466 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:41.881328 9466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:41.888931 9471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:41.888949 9474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:41.889055 9466 server_base.cc:1048] running on GCE node
W20250628 20:03:41.888929 9472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:41.889571 9466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:41.890202 9466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:41.893447 9466 hybrid_clock.cc:648] HybridClock initialized: now 1751141021893429 us; error 30 us; skew 500 ppm
I20250628 20:03:41.895417 9466 webserver.cc:469] Webserver started at http://127.5.100.66:35087/ using document root <none> and password file <none>
I20250628 20:03:41.896004 9466 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:41.896059 9466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:41.896237 9466 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:41.897881 9466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "496c40dbb5cc4b1e83544c3e59d0d527"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.898321 9466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "496c40dbb5cc4b1e83544c3e59d0d527"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.901582 9466 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:41.903815 9480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:41.904825 9466 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:41.904938 9466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "496c40dbb5cc4b1e83544c3e59d0d527"
format_stamp: "Formatted at 2025-06-28 20:03:41 on dist-test-slave-rmcj"
I20250628 20:03:41.905027 9466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:41.920720 9466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:41.921401 9466 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:41.921567 9466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:41.922178 9466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:41.923156 9466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:41.923203 9466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:41.923246 9466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:41.923274 9466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:41.931831 9466 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:36283
I20250628 20:03:41.931882 9593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:36283 every 8 connection(s)
I20250628 20:03:41.932826 9466 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 20:03:41.933240 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 9466
I20250628 20:03:41.933343 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 20:03:41.935662 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:0
--local_ip_for_outbound_sockets=127.5.100.67
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--builtin_ntp_servers=127.5.100.84:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 20:03:41.942705 9594 heartbeater.cc:344] Connected to a master server at 127.5.100.126:39089
I20250628 20:03:41.943013 9594 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:41.943727 9594 heartbeater.cc:507] Master 127.5.100.126:39089 requested a full tablet report, sending...
I20250628 20:03:41.944900 9272 ts_manager.cc:194] Registered new tserver with Master: 496c40dbb5cc4b1e83544c3e59d0d527 (127.5.100.66:36283)
I20250628 20:03:41.945647 9272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:37107
W20250628 20:03:42.040997 9597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:42.041227 9597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:42.041270 9597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:42.044374 9597 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 20:03:42.044438 9597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:42.044504 9597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:42.048125 9597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--never_fsync=true
--heap_profile_path=/tmp/kudu.9597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:42.049172 9597 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:42.050272 9597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:42.057643 9602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:42.057654 9603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:42.057664 9605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:42.057791 9597 server_base.cc:1048] running on GCE node
I20250628 20:03:42.058223 9597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:42.058745 9597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:42.059895 9597 hybrid_clock.cc:648] HybridClock initialized: now 1751141022059876 us; error 29 us; skew 500 ppm
I20250628 20:03:42.061626 9597 webserver.cc:469] Webserver started at http://127.5.100.67:44731/ using document root <none> and password file <none>
I20250628 20:03:42.062251 9597 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:42.062327 9597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:42.062500 9597 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:42.064028 9597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3420ebb3a0d640f4b7a348713f20af89"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.064467 9597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3420ebb3a0d640f4b7a348713f20af89"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.067591 9597 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250628 20:03:42.069531 9611 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.070585 9597 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:42.070686 9597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3420ebb3a0d640f4b7a348713f20af89"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.070765 9597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:42.087847 9597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:42.088474 9597 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:42.088625 9597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:42.089154 9597 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:42.090097 9597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:42.090144 9597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.090195 9597 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:42.090220 9597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.098644 9597 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:40727
I20250628 20:03:42.098672 9724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:40727 every 8 connection(s)
I20250628 20:03:42.099550 9597 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 20:03:42.101814 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 9597
I20250628 20:03:42.101925 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 20:03:42.104015 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.68:0
--local_ip_for_outbound_sockets=127.5.100.68
--webserver_interface=127.5.100.68
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--builtin_ntp_servers=127.5.100.84:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 20:03:42.108471 9725 heartbeater.cc:344] Connected to a master server at 127.5.100.126:39089
I20250628 20:03:42.108698 9725 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:42.109258 9725 heartbeater.cc:507] Master 127.5.100.126:39089 requested a full tablet report, sending...
I20250628 20:03:42.110234 9272 ts_manager.cc:194] Registered new tserver with Master: 3420ebb3a0d640f4b7a348713f20af89 (127.5.100.67:40727)
I20250628 20:03:42.110754 9272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:42007
W20250628 20:03:42.209889 9728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:42.210111 9728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:42.210179 9728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:42.213325 9728 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 20:03:42.213402 9728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:42.213498 9728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.68
I20250628 20:03:42.217201 9728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.100.68
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--never_fsync=true
--heap_profile_path=/tmp/kudu.9728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.68
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:42.218303 9728 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:42.219465 9728 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:42.227227 9736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:42.227367 9728 server_base.cc:1048] running on GCE node
W20250628 20:03:42.227200 9733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:42.227177 9734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:42.227859 9728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:42.228425 9728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:42.229617 9728 hybrid_clock.cc:648] HybridClock initialized: now 1751141022229596 us; error 39 us; skew 500 ppm
I20250628 20:03:42.231413 9728 webserver.cc:469] Webserver started at http://127.5.100.68:35279/ using document root <none> and password file <none>
I20250628 20:03:42.231976 9728 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:42.232031 9728 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:42.232214 9728 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:42.233834 9728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.234294 9728 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.237526 9728 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:42.239624 9742 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.240597 9728 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:42.240694 9728 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.240773 9728 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:42.254972 9728 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:42.255590 9728 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:42.255745 9728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:42.256327 9728 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:42.257273 9728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:42.257320 9728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.257372 9728 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:42.257400 9728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.266160 9728 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.68:43929
I20250628 20:03:42.266198 9855 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.68:43929 every 8 connection(s)
I20250628 20:03:42.267099 9728 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250628 20:03:42.269305 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 9728
I20250628 20:03:42.269423 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250628 20:03:42.271464 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.69:0
--local_ip_for_outbound_sockets=127.5.100.69
--webserver_interface=127.5.100.69
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--builtin_ntp_servers=127.5.100.84:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 20:03:42.276149 9856 heartbeater.cc:344] Connected to a master server at 127.5.100.126:39089
I20250628 20:03:42.276424 9856 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:42.277125 9856 heartbeater.cc:507] Master 127.5.100.126:39089 requested a full tablet report, sending...
I20250628 20:03:42.278199 9272 ts_manager.cc:194] Registered new tserver with Master: 0c1940eb16e94c3a82bf6198ba7c3b5e (127.5.100.68:43929)
I20250628 20:03:42.278789 9272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.68:57581
W20250628 20:03:42.375959 9859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:42.376184 9859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:42.376247 9859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:42.379438 9859 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 20:03:42.379513 9859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:42.379601 9859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.69
I20250628 20:03:42.383329 9859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.100.69
--webserver_port=0
--tserver_master_addrs=127.5.100.126:39089
--never_fsync=true
--heap_profile_path=/tmp/kudu.9859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.69
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:42.384382 9859 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:42.385499 9859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:42.392704 9867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:42.392724 9864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:42.392715 9865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:42.392978 9859 server_base.cc:1048] running on GCE node
I20250628 20:03:42.393340 9859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:42.393898 9859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:42.395059 9859 hybrid_clock.cc:648] HybridClock initialized: now 1751141022395035 us; error 33 us; skew 500 ppm
I20250628 20:03:42.396816 9859 webserver.cc:469] Webserver started at http://127.5.100.69:38217/ using document root <none> and password file <none>
I20250628 20:03:42.397372 9859 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:42.397423 9859 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:42.397604 9859 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:42.399451 9859 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f10a2b1c7fc345a78fb66d89c607de59"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.400044 9859 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f10a2b1c7fc345a78fb66d89c607de59"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.403424 9859 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:42.405527 9873 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.406570 9859 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:42.406672 9859 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f10a2b1c7fc345a78fb66d89c607de59"
format_stamp: "Formatted at 2025-06-28 20:03:42 on dist-test-slave-rmcj"
I20250628 20:03:42.406760 9859 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:42.432483 9859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:42.433199 9859 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:42.433367 9859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:42.434016 9859 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:42.434962 9859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:42.435009 9859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.435061 9859 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:42.435097 9859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:42.443593 9859 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.69:39249
I20250628 20:03:42.443641 9986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.69:39249 every 8 connection(s)
I20250628 20:03:42.444531 9859 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250628 20:03:42.447144 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 9859
I20250628 20:03:42.447263 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250628 20:03:42.453199 9987 heartbeater.cc:344] Connected to a master server at 127.5.100.126:39089
I20250628 20:03:42.453433 9987 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:42.453984 9987 heartbeater.cc:507] Master 127.5.100.126:39089 requested a full tablet report, sending...
I20250628 20:03:42.454880 9272 ts_manager.cc:194] Registered new tserver with Master: f10a2b1c7fc345a78fb66d89c607de59 (127.5.100.69:39249)
I20250628 20:03:42.455435 9272 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.69:55327
I20250628 20:03:42.461292 5521 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250628 20:03:42.469650 9272 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44548:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 20:03:42.487704 9659 tablet_service.cc:1468] Processing CreateTablet for tablet 501b28d1576f40f2998e825d3b0ca81c (DEFAULT_TABLE table=TestTable [id=61d1b3f5d9a741ba986383ec255c825b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:42.487720 9528 tablet_service.cc:1468] Processing CreateTablet for tablet 501b28d1576f40f2998e825d3b0ca81c (DEFAULT_TABLE table=TestTable [id=61d1b3f5d9a741ba986383ec255c825b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:42.487720 9790 tablet_service.cc:1468] Processing CreateTablet for tablet 501b28d1576f40f2998e825d3b0ca81c (DEFAULT_TABLE table=TestTable [id=61d1b3f5d9a741ba986383ec255c825b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:42.488600 9659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501b28d1576f40f2998e825d3b0ca81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:42.488617 9528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501b28d1576f40f2998e825d3b0ca81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:42.488617 9790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501b28d1576f40f2998e825d3b0ca81c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:42.493427 10004 tablet_bootstrap.cc:492] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89: Bootstrap starting.
I20250628 20:03:42.495121 10006 tablet_bootstrap.cc:492] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e: Bootstrap starting.
I20250628 20:03:42.495642 10004 tablet_bootstrap.cc:654] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:42.496342 10004 log.cc:826] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:42.496869 10006 tablet_bootstrap.cc:654] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:42.497499 10006 log.cc:826] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:42.497910 10004 tablet_bootstrap.cc:492] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89: No bootstrap required, opened a new log
I20250628 20:03:42.498109 10004 ts_tablet_manager.cc:1397] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250628 20:03:42.498409 10005 tablet_bootstrap.cc:492] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527: Bootstrap starting.
I20250628 20:03:42.499327 10006 tablet_bootstrap.cc:492] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e: No bootstrap required, opened a new log
I20250628 20:03:42.499522 10006 ts_tablet_manager.cc:1397] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250628 20:03:42.500619 10005 tablet_bootstrap.cc:654] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:42.500758 10004 raft_consensus.cc:357] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.500947 10004 raft_consensus.cc:383] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:42.500998 10004 raft_consensus.cc:738] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3420ebb3a0d640f4b7a348713f20af89, State: Initialized, Role: FOLLOWER
I20250628 20:03:42.501406 10004 consensus_queue.cc:260] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.501426 10005 log.cc:826] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:42.502104 9725 heartbeater.cc:499] Master 127.5.100.126:39089 was elected leader, sending a full tablet report...
I20250628 20:03:42.502233 10004 ts_tablet_manager.cc:1428] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 20:03:42.502489 10006 raft_consensus.cc:357] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.502692 10006 raft_consensus.cc:383] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:42.502753 10006 raft_consensus.cc:738] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1940eb16e94c3a82bf6198ba7c3b5e, State: Initialized, Role: FOLLOWER
I20250628 20:03:42.503232 10006 consensus_queue.cc:260] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.503938 10005 tablet_bootstrap.cc:492] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527: No bootstrap required, opened a new log
I20250628 20:03:42.504096 10006 ts_tablet_manager.cc:1428] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250628 20:03:42.504128 10005 ts_tablet_manager.cc:1397] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250628 20:03:42.504426 9856 heartbeater.cc:499] Master 127.5.100.126:39089 was elected leader, sending a full tablet report...
I20250628 20:03:42.506994 10005 raft_consensus.cc:357] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.507181 10005 raft_consensus.cc:383] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:42.507237 10005 raft_consensus.cc:738] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496c40dbb5cc4b1e83544c3e59d0d527, State: Initialized, Role: FOLLOWER
I20250628 20:03:42.507594 10005 consensus_queue.cc:260] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.508195 9594 heartbeater.cc:499] Master 127.5.100.126:39089 was elected leader, sending a full tablet report...
I20250628 20:03:42.508363 10005 ts_tablet_manager.cc:1428] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250628 20:03:42.517871 9857 tablet.cc:2378] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 20:03:42.600576 9726 tablet.cc:2378] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:42.656558 10011 raft_consensus.cc:491] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:42.656769 10011 raft_consensus.cc:513] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.658763 10011 leader_election.cc:290] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3420ebb3a0d640f4b7a348713f20af89 (127.5.100.67:40727), 496c40dbb5cc4b1e83544c3e59d0d527 (127.5.100.66:36283)
I20250628 20:03:42.661861 9679 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b28d1576f40f2998e825d3b0ca81c" candidate_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3420ebb3a0d640f4b7a348713f20af89" is_pre_election: true
I20250628 20:03:42.662014 9548 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b28d1576f40f2998e825d3b0ca81c" candidate_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" is_pre_election: true
I20250628 20:03:42.662179 9679 raft_consensus.cc:2466] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c1940eb16e94c3a82bf6198ba7c3b5e in term 0.
I20250628 20:03:42.662289 9548 raft_consensus.cc:2466] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c1940eb16e94c3a82bf6198ba7c3b5e in term 0.
I20250628 20:03:42.662631 9745 leader_election.cc:304] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1940eb16e94c3a82bf6198ba7c3b5e, 3420ebb3a0d640f4b7a348713f20af89; no voters:
I20250628 20:03:42.662910 10011 raft_consensus.cc:2802] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 20:03:42.662982 10011 raft_consensus.cc:491] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 20:03:42.663031 10011 raft_consensus.cc:3058] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:42.663866 10011 raft_consensus.cc:513] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.664217 10011 leader_election.cc:290] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [CANDIDATE]: Term 1 election: Requested vote from peers 3420ebb3a0d640f4b7a348713f20af89 (127.5.100.67:40727), 496c40dbb5cc4b1e83544c3e59d0d527 (127.5.100.66:36283)
I20250628 20:03:42.664583 9548 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b28d1576f40f2998e825d3b0ca81c" candidate_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "496c40dbb5cc4b1e83544c3e59d0d527"
I20250628 20:03:42.664583 9679 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501b28d1576f40f2998e825d3b0ca81c" candidate_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3420ebb3a0d640f4b7a348713f20af89"
I20250628 20:03:42.664706 9548 raft_consensus.cc:3058] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:42.664706 9679 raft_consensus.cc:3058] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:42.665771 9679 raft_consensus.cc:2466] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c1940eb16e94c3a82bf6198ba7c3b5e in term 1.
I20250628 20:03:42.665771 9548 raft_consensus.cc:2466] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c1940eb16e94c3a82bf6198ba7c3b5e in term 1.
I20250628 20:03:42.666056 9745 leader_election.cc:304] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c1940eb16e94c3a82bf6198ba7c3b5e, 3420ebb3a0d640f4b7a348713f20af89, 496c40dbb5cc4b1e83544c3e59d0d527; no voters:
I20250628 20:03:42.666234 10011 raft_consensus.cc:2802] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:42.666479 10011 raft_consensus.cc:695] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [term 1 LEADER]: Becoming Leader. State: Replica: 0c1940eb16e94c3a82bf6198ba7c3b5e, State: Running, Role: LEADER
I20250628 20:03:42.666806 10011 consensus_queue.cc:237] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } }
I20250628 20:03:42.669731 9272 catalog_manager.cc:5582] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1940eb16e94c3a82bf6198ba7c3b5e (127.5.100.68). New cstate: current_term: 1 leader_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1940eb16e94c3a82bf6198ba7c3b5e" member_type: VOTER last_known_addr { host: "127.5.100.68" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "496c40dbb5cc4b1e83544c3e59d0d527" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 36283 } health_report { overall_health: UNKNOWN } } }
I20250628 20:03:42.676306 5521 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250628 20:03:42.677577 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 496c40dbb5cc4b1e83544c3e59d0d527 to finish bootstrapping
I20250628 20:03:42.682837 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3420ebb3a0d640f4b7a348713f20af89 to finish bootstrapping
W20250628 20:03:42.684273 9595 tablet.cc:2378] T 501b28d1576f40f2998e825d3b0ca81c P 496c40dbb5cc4b1e83544c3e59d0d527: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:42.687937 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0c1940eb16e94c3a82bf6198ba7c3b5e to finish bootstrapping
I20250628 20:03:42.692389 5521 test_util.cc:276] Using random seed: 1251734562
I20250628 20:03:42.700158 5521 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250628 20:03:42.700337 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 9466
W20250628 20:03:42.714031 9744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.100.66:36283: connect: Connection refused (error 111)
I20250628 20:03:42.714363 9679 raft_consensus.cc:1273] T 501b28d1576f40f2998e825d3b0ca81c P 3420ebb3a0d640f4b7a348713f20af89 [term 1 FOLLOWER]: Refusing update from remote peer 0c1940eb16e94c3a82bf6198ba7c3b5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250628 20:03:42.714932 9744 consensus_peers.cc:489] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e -> Peer 496c40dbb5cc4b1e83544c3e59d0d527 (127.5.100.66:36283): Couldn't send request to peer 496c40dbb5cc4b1e83544c3e59d0d527. Status: Network error: Client connection negotiation failed: client connection to 127.5.100.66:36283: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 20:03:42.715005 10014 consensus_queue.cc:1035] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3420ebb3a0d640f4b7a348713f20af89" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:42.722579 10027 mvcc.cc:204] Tried to move back new op lower bound from 7172673629030690816 to 7172673628845633536. Current Snapshot: MvccSnapshot[applied={T|T < 7172673629030690816}]
I20250628 20:03:42.770417 9463 heartbeater.cc:499] Master 127.5.100.126:39089 was elected leader, sending a full tablet report...
I20250628 20:03:42.937026 9790 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:42.940542 9921 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:42.940882 9397 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:42.941423 9659 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:43.457438 9987 heartbeater.cc:499] Master 127.5.100.126:39089 was elected leader, sending a full tablet report...
I20250628 20:03:44.040373 9790 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:44.041769 9397 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:44.042502 9921 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:44.050431 9659 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:45.136670 9659 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:45.137656 9397 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:45.142235 9790 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:45.152016 9921 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250628 20:03:45.389842 9744 consensus_peers.cc:489] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e -> Peer 496c40dbb5cc4b1e83544c3e59d0d527 (127.5.100.66:36283): Couldn't send request to peer 496c40dbb5cc4b1e83544c3e59d0d527. Status: Network error: Client connection negotiation failed: client connection to 127.5.100.66:36283: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250628 20:03:46.229103 9397 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:46.230923 9921 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:46.232467 9790 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:46.240868 9659 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:47.332011 9659 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:47.334964 9790 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:47.346254 9921 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:47.359149 9397 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250628 20:03:48.078086 9744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.100.66:36283: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250628 20:03:48.078935 9744 consensus_peers.cc:489] T 501b28d1576f40f2998e825d3b0ca81c P 0c1940eb16e94c3a82bf6198ba7c3b5e -> Peer 496c40dbb5cc4b1e83544c3e59d0d527 (127.5.100.66:36283): Couldn't send request to peer 496c40dbb5cc4b1e83544c3e59d0d527. Status: Network error: Client connection negotiation failed: client connection to 127.5.100.66:36283: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250628 20:03:48.388240 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 9335
I20250628 20:03:48.394069 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 9597
I20250628 20:03:48.400512 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 9728
I20250628 20:03:48.405932 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 9859
I20250628 20:03:48.410763 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 9244
2025-06-28T20:03:48Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (7071 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (7071 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250628 20:03:48.432130 5521 test_util.cc:276] Using random seed: 1257474298
I20250628 20:03:48.432878 5521 ts_itest-base.cc:115] Starting cluster with:
I20250628 20:03:48.432924 5521 ts_itest-base.cc:116] --------------
I20250628 20:03:48.432940 5521 ts_itest-base.cc:117] 1 tablet servers
I20250628 20:03:48.432950 5521 ts_itest-base.cc:118] 1 replicas per TS
I20250628 20:03:48.432958 5521 ts_itest-base.cc:119] --------------
2025-06-28T20:03:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T20:03:48Z Disabled control of system clock
I20250628 20:03:48.441026 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:41341
--webserver_interface=127.5.100.126
--webserver_port=0
--builtin_ntp_servers=127.5.100.84:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:41341 with env {}
W20250628 20:03:48.543175 10210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:48.543402 10210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:48.543457 10210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:48.546572 10210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:48.546631 10210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:48.546659 10210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:48.546680 10210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:48.550309 10210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:41341
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:41341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:48.551364 10210 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:48.552510 10210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:48.559036 10216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:48.559041 10215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:48.559039 10218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:48.559381 10210 server_base.cc:1048] running on GCE node
I20250628 20:03:48.559875 10210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:48.560755 10210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:48.561904 10210 hybrid_clock.cc:648] HybridClock initialized: now 1751141028561892 us; error 31 us; skew 500 ppm
I20250628 20:03:48.568033 10210 webserver.cc:469] Webserver started at http://127.5.100.126:38927/ using document root <none> and password file <none>
I20250628 20:03:48.568629 10210 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:48.568688 10210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:48.568878 10210 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:48.570562 10210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed"
format_stamp: "Formatted at 2025-06-28 20:03:48 on dist-test-slave-rmcj"
I20250628 20:03:48.571027 10210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed"
format_stamp: "Formatted at 2025-06-28 20:03:48 on dist-test-slave-rmcj"
I20250628 20:03:48.574344 10210 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:48.576607 10224 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:48.577617 10210 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:48.577778 10210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed"
format_stamp: "Formatted at 2025-06-28 20:03:48 on dist-test-slave-rmcj"
I20250628 20:03:48.577880 10210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:48.591473 10210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:48.592164 10210 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:48.592339 10210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:48.599409 10276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:41341 every 8 connection(s)
I20250628 20:03:48.599411 10210 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:41341
I20250628 20:03:48.600471 10210 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 20:03:48.603049 10277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:48.605962 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 10210
I20250628 20:03:48.606079 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 20:03:48.608554 10277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed: Bootstrap starting.
I20250628 20:03:48.610874 10277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:48.611747 10277 log.cc:826] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:48.613969 10277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed: No bootstrap required, opened a new log
I20250628 20:03:48.616267 10277 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 41341 } }
I20250628 20:03:48.616453 10277 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:48.616498 10277 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4a2e3c662a476faefa8aaf9dcab1ed, State: Initialized, Role: FOLLOWER
I20250628 20:03:48.616900 10277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 41341 } }
I20250628 20:03:48.617014 10277 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:48.617072 10277 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:48.617177 10277 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:48.617983 10277 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 41341 } }
I20250628 20:03:48.618302 10277 leader_election.cc:304] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b4a2e3c662a476faefa8aaf9dcab1ed; no voters:
I20250628 20:03:48.618559 10277 leader_election.cc:290] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:48.618700 10282 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:48.618888 10282 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [term 1 LEADER]: Becoming Leader. State: Replica: 9b4a2e3c662a476faefa8aaf9dcab1ed, State: Running, Role: LEADER
I20250628 20:03:48.619158 10282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 41341 } }
I20250628 20:03:48.619585 10277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:48.620940 10284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b4a2e3c662a476faefa8aaf9dcab1ed. Latest consensus state: current_term: 1 leader_uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 41341 } } }
I20250628 20:03:48.621155 10283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4a2e3c662a476faefa8aaf9dcab1ed" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 41341 } } }
I20250628 20:03:48.621246 10283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:48.621168 10284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed [sys.catalog]: This master's current role is: LEADER
W20250628 20:03:48.624086 10297 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250628 20:03:48.624147 10297 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250628 20:03:48.624219 10295 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:48.625066 10295 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:48.630014 10295 catalog_manager.cc:1349] Generated new cluster ID: 6cc496b1a9b24b8682fc2df71dca6d22
I20250628 20:03:48.630087 10295 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:48.649813 10295 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:48.650784 10295 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:48.659111 10295 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9b4a2e3c662a476faefa8aaf9dcab1ed: Generated new TSK 0
I20250628 20:03:48.660006 10295 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:48.668335 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:0
--local_ip_for_outbound_sockets=127.5.100.65
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:41341
--builtin_ntp_servers=127.5.100.84:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 20:03:48.771688 10301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:48.771907 10301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:48.771951 10301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:48.775158 10301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:48.775271 10301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:48.779079 10301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:41341
--never_fsync=true
--heap_profile_path=/tmp/kudu.10301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:48.780136 10301 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:48.781275 10301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:48.788913 10307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:48.788944 10309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:48.788916 10306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:48.789206 10301 server_base.cc:1048] running on GCE node
I20250628 20:03:48.789578 10301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:48.790179 10301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:48.791318 10301 hybrid_clock.cc:648] HybridClock initialized: now 1751141028791297 us; error 33 us; skew 500 ppm
I20250628 20:03:48.793169 10301 webserver.cc:469] Webserver started at http://127.5.100.65:33059/ using document root <none> and password file <none>
I20250628 20:03:48.793766 10301 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:48.793824 10301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:48.794021 10301 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:48.795567 10301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2c96dc3328ef412694f82767d1746e63"
format_stamp: "Formatted at 2025-06-28 20:03:48 on dist-test-slave-rmcj"
I20250628 20:03:48.796015 10301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2c96dc3328ef412694f82767d1746e63"
format_stamp: "Formatted at 2025-06-28 20:03:48 on dist-test-slave-rmcj"
I20250628 20:03:48.799207 10301 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250628 20:03:48.801239 10315 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:48.802256 10301 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:48.802356 10301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2c96dc3328ef412694f82767d1746e63"
format_stamp: "Formatted at 2025-06-28 20:03:48 on dist-test-slave-rmcj"
I20250628 20:03:48.802438 10301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:48.819115 10301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:48.819758 10301 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:48.819928 10301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:48.820464 10301 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:48.821441 10301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:48.821489 10301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:48.821597 10301 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:48.821632 10301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:48.829871 10301 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:44547
I20250628 20:03:48.829923 10428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:44547 every 8 connection(s)
I20250628 20:03:48.830798 10301 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 20:03:48.834077 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 10301
I20250628 20:03:48.834190 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751141006956821-5521-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 20:03:48.839465 10429 heartbeater.cc:344] Connected to a master server at 127.5.100.126:41341
I20250628 20:03:48.839686 10429 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:48.840184 10429 heartbeater.cc:507] Master 127.5.100.126:41341 requested a full tablet report, sending...
I20250628 20:03:48.841446 10241 ts_manager.cc:194] Registered new tserver with Master: 2c96dc3328ef412694f82767d1746e63 (127.5.100.65:44547)
I20250628 20:03:48.842558 10241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:59121
I20250628 20:03:48.847777 5521 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250628 20:03:48.855547 10241 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42028:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 20:03:48.871927 10363 tablet_service.cc:1468] Processing CreateTablet for tablet a9d3f956d5fb45feb3d989712ec82584 (DEFAULT_TABLE table=TestTable [id=4df453d455d7422b99bd804997b876a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:48.872933 10363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d3f956d5fb45feb3d989712ec82584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:48.878095 10439 tablet_bootstrap.cc:492] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63: Bootstrap starting.
I20250628 20:03:48.881579 10439 tablet_bootstrap.cc:654] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:48.882580 10439 log.cc:826] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:48.884583 10439 tablet_bootstrap.cc:492] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63: No bootstrap required, opened a new log
I20250628 20:03:48.884794 10439 ts_tablet_manager.cc:1397] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250628 20:03:48.887254 10439 raft_consensus.cc:357] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c96dc3328ef412694f82767d1746e63" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44547 } }
I20250628 20:03:48.887418 10439 raft_consensus.cc:383] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:48.887455 10439 raft_consensus.cc:738] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c96dc3328ef412694f82767d1746e63, State: Initialized, Role: FOLLOWER
I20250628 20:03:48.887830 10439 consensus_queue.cc:260] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c96dc3328ef412694f82767d1746e63" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44547 } }
I20250628 20:03:48.887943 10439 raft_consensus.cc:397] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:48.887981 10439 raft_consensus.cc:491] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:48.888054 10439 raft_consensus.cc:3058] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:48.888777 10439 raft_consensus.cc:513] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c96dc3328ef412694f82767d1746e63" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44547 } }
I20250628 20:03:48.889086 10439 leader_election.cc:304] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c96dc3328ef412694f82767d1746e63; no voters:
I20250628 20:03:48.889389 10439 leader_election.cc:290] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:48.889585 10441 raft_consensus.cc:2802] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:48.889770 10441 raft_consensus.cc:695] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [term 1 LEADER]: Becoming Leader. State: Replica: 2c96dc3328ef412694f82767d1746e63, State: Running, Role: LEADER
I20250628 20:03:48.890043 10441 consensus_queue.cc:237] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c96dc3328ef412694f82767d1746e63" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44547 } }
I20250628 20:03:48.890257 10439 ts_tablet_manager.cc:1428] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250628 20:03:48.890616 10429 heartbeater.cc:499] Master 127.5.100.126:41341 was elected leader, sending a full tablet report...
I20250628 20:03:48.893422 10241 catalog_manager.cc:5582] T a9d3f956d5fb45feb3d989712ec82584 P 2c96dc3328ef412694f82767d1746e63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c96dc3328ef412694f82767d1746e63 (127.5.100.65). New cstate: current_term: 1 leader_uuid: "2c96dc3328ef412694f82767d1746e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c96dc3328ef412694f82767d1746e63" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 44547 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:48.899830 5521 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250628 20:03:48.900996 5521 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2c96dc3328ef412694f82767d1746e63 to finish bootstrapping
I20250628 20:03:49.040732 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 10301
I20250628 20:03:49.045966 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 10210
2025-06-28T20:03:49Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (632 ms)
[----------] 1 test from ListTableCliSimpleParamTest (632 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250628 20:03:49.064465 5521 test_util.cc:276] Using random seed: 1258106637
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (1 ms)
[----------] 1 test from ListTableCliParamTest (1 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-28T20:03:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T20:03:49Z Disabled control of system clock
I20250628 20:03:49.073904 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:35675
--webserver_interface=127.5.100.126
--webserver_port=0
--builtin_ntp_servers=127.5.100.84:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:35675 with env {}
W20250628 20:03:49.176318 10463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:49.176561 10463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:49.176609 10463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:49.179878 10463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:49.179951 10463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:49.179971 10463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:49.179988 10463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:49.184065 10463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:35675
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:35675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:49.185382 10463 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:49.186678 10463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:49.193243 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:49.193383 10463 server_base.cc:1048] running on GCE node
W20250628 20:03:49.193244 10468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:49.193557 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:49.194070 10463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:49.195024 10463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:49.196163 10463 hybrid_clock.cc:648] HybridClock initialized: now 1751141029196150 us; error 30 us; skew 500 ppm
I20250628 20:03:49.198041 10463 webserver.cc:469] Webserver started at http://127.5.100.126:46205/ using document root <none> and password file <none>
I20250628 20:03:49.198617 10463 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:49.198676 10463 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:49.198851 10463 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:49.200415 10463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/instance:
uuid: "f0bfcc775367479ca935573d561fa982"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.200865 10463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal/instance:
uuid: "f0bfcc775367479ca935573d561fa982"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.204128 10463 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250628 20:03:49.206229 10477 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.207222 10463 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:49.207329 10463 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
uuid: "f0bfcc775367479ca935573d561fa982"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.207412 10463 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:49.254184 10463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:49.254891 10463 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:49.255069 10463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:49.262111 10463 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:35675
I20250628 20:03:49.262166 10529 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:35675 every 8 connection(s)
I20250628 20:03:49.263155 10463 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/info.pb
I20250628 20:03:49.265810 10530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:49.268987 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 10463
I20250628 20:03:49.269109 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal/instance
I20250628 20:03:49.271405 10530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982: Bootstrap starting.
I20250628 20:03:49.273594 10530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:49.274334 10530 log.cc:826] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:49.276197 10530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982: No bootstrap required, opened a new log
I20250628 20:03:49.278900 10530 raft_consensus.cc:357] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bfcc775367479ca935573d561fa982" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } }
I20250628 20:03:49.279085 10530 raft_consensus.cc:383] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:49.279135 10530 raft_consensus.cc:738] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0bfcc775367479ca935573d561fa982, State: Initialized, Role: FOLLOWER
I20250628 20:03:49.279518 10530 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bfcc775367479ca935573d561fa982" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } }
I20250628 20:03:49.279628 10530 raft_consensus.cc:397] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:49.279685 10530 raft_consensus.cc:491] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:49.279783 10530 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:49.280547 10530 raft_consensus.cc:513] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bfcc775367479ca935573d561fa982" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } }
I20250628 20:03:49.280889 10530 leader_election.cc:304] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0bfcc775367479ca935573d561fa982; no voters:
I20250628 20:03:49.281164 10530 leader_election.cc:290] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:49.281327 10535 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:49.281539 10535 raft_consensus.cc:695] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [term 1 LEADER]: Becoming Leader. State: Replica: f0bfcc775367479ca935573d561fa982, State: Running, Role: LEADER
I20250628 20:03:49.281868 10535 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bfcc775367479ca935573d561fa982" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } }
I20250628 20:03:49.282269 10530 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:49.283561 10536 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0bfcc775367479ca935573d561fa982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bfcc775367479ca935573d561fa982" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } } }
I20250628 20:03:49.283685 10536 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:49.283953 10537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0bfcc775367479ca935573d561fa982. Latest consensus state: current_term: 1 leader_uuid: "f0bfcc775367479ca935573d561fa982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0bfcc775367479ca935573d561fa982" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } } }
I20250628 20:03:49.284041 10537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982 [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:49.284518 10544 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:49.287086 10544 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:49.292467 10544 catalog_manager.cc:1349] Generated new cluster ID: 98f0f2c84ce24829a293cebbbe784839
I20250628 20:03:49.292548 10544 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:49.312968 10544 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:49.313947 10544 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:49.323220 10544 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f0bfcc775367479ca935573d561fa982: Generated new TSK 0
I20250628 20:03:49.323920 10544 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 20:03:49.332679 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:0
--local_ip_for_outbound_sockets=127.5.100.65
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:35675
--builtin_ntp_servers=127.5.100.84:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250628 20:03:49.436372 10554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:49.436595 10554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:49.436640 10554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:49.439908 10554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:49.440027 10554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:49.443822 10554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=0
--tserver_master_addrs=127.5.100.126:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.10554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:49.444945 10554 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:49.446146 10554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:49.453778 10560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:49.453800 10559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:49.453894 10554 server_base.cc:1048] running on GCE node
W20250628 20:03:49.453750 10562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:49.454334 10554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:49.454922 10554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:49.456080 10554 hybrid_clock.cc:648] HybridClock initialized: now 1751141029456060 us; error 29 us; skew 500 ppm
I20250628 20:03:49.458130 10554 webserver.cc:469] Webserver started at http://127.5.100.65:46597/ using document root <none> and password file <none>
I20250628 20:03:49.458688 10554 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:49.458740 10554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:49.458922 10554 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:49.460563 10554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/instance:
uuid: "81ddd3063d4842e790d3a80c6f0d4f7c"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.461006 10554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal/instance:
uuid: "81ddd3063d4842e790d3a80c6f0d4f7c"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.464375 10554 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250628 20:03:49.466583 10568 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.467628 10554 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:49.467742 10554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
uuid: "81ddd3063d4842e790d3a80c6f0d4f7c"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.467840 10554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:49.493499 10554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:49.494231 10554 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:49.494405 10554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:49.495021 10554 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:49.496006 10554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:49.496054 10554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.496099 10554 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:49.496129 10554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.504942 10554 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:38815
I20250628 20:03:49.504966 10681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:38815 every 8 connection(s)
I20250628 20:03:49.505983 10554 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/info.pb
I20250628 20:03:49.509032 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 10554
I20250628 20:03:49.509181 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal/instance
I20250628 20:03:49.511482 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:0
--local_ip_for_outbound_sockets=127.5.100.66
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:35675
--builtin_ntp_servers=127.5.100.84:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 20:03:49.516026 10682 heartbeater.cc:344] Connected to a master server at 127.5.100.126:35675
I20250628 20:03:49.516291 10682 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:49.516868 10682 heartbeater.cc:507] Master 127.5.100.126:35675 requested a full tablet report, sending...
I20250628 20:03:49.518364 10494 ts_manager.cc:194] Registered new tserver with Master: 81ddd3063d4842e790d3a80c6f0d4f7c (127.5.100.65:38815)
I20250628 20:03:49.519589 10494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:53695
W20250628 20:03:49.614995 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:49.615226 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:49.615288 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:49.618443 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:49.618567 10685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:49.622275 10685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=0
--tserver_master_addrs=127.5.100.126:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.10685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:49.623365 10685 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:49.624500 10685 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:49.631923 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:49.631969 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:49.631968 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:49.632159 10685 server_base.cc:1048] running on GCE node
I20250628 20:03:49.632544 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:49.633126 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:49.634260 10685 hybrid_clock.cc:648] HybridClock initialized: now 1751141029634241 us; error 39 us; skew 500 ppm
I20250628 20:03:49.636057 10685 webserver.cc:469] Webserver started at http://127.5.100.66:34971/ using document root <none> and password file <none>
I20250628 20:03:49.636610 10685 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:49.636682 10685 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:49.636884 10685 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:49.638478 10685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/instance:
uuid: "fca3462212d04f1aafd36a29e9b20ec6"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.638928 10685 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal/instance:
uuid: "fca3462212d04f1aafd36a29e9b20ec6"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.642163 10685 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:49.644294 10699 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.645285 10685 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:49.645390 10685 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
uuid: "fca3462212d04f1aafd36a29e9b20ec6"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.645473 10685 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:49.658190 10685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:49.658836 10685 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:49.659004 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:49.659543 10685 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:49.660480 10685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:49.660526 10685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.660579 10685 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:49.660602 10685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.668962 10685 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:44981
I20250628 20:03:49.668979 10812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:44981 every 8 connection(s)
I20250628 20:03:49.669946 10685 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/info.pb
I20250628 20:03:49.677313 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 10685
I20250628 20:03:49.677433 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal/instance
I20250628 20:03:49.678498 10813 heartbeater.cc:344] Connected to a master server at 127.5.100.126:35675
I20250628 20:03:49.678737 10813 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:49.679270 10813 heartbeater.cc:507] Master 127.5.100.126:35675 requested a full tablet report, sending...
I20250628 20:03:49.679544 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:0
--local_ip_for_outbound_sockets=127.5.100.67
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:35675
--builtin_ntp_servers=127.5.100.84:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 20:03:49.680150 10494 ts_manager.cc:194] Registered new tserver with Master: fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981)
I20250628 20:03:49.680682 10494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:54465
W20250628 20:03:49.782650 10816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:49.782897 10816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:49.782950 10816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:49.786227 10816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:49.786338 10816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:49.790025 10816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=0
--tserver_master_addrs=127.5.100.126:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.10816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:49.791072 10816 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:49.792264 10816 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:49.799769 10824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:49.799754 10821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:49.799938 10816 server_base.cc:1048] running on GCE node
W20250628 20:03:49.799759 10822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:49.800392 10816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:49.800957 10816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:49.802102 10816 hybrid_clock.cc:648] HybridClock initialized: now 1751141029802085 us; error 30 us; skew 500 ppm
I20250628 20:03:49.803926 10816 webserver.cc:469] Webserver started at http://127.5.100.67:44775/ using document root <none> and password file <none>
I20250628 20:03:49.804484 10816 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:49.804543 10816 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:49.804716 10816 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:49.806336 10816 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/instance:
uuid: "88a585f79fd34be99b0e3b0c0c0faf15"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.806782 10816 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal/instance:
uuid: "88a585f79fd34be99b0e3b0c0c0faf15"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.810004 10816 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:49.812206 10830 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.813199 10816 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:49.813304 10816 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
uuid: "88a585f79fd34be99b0e3b0c0c0faf15"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:49.813390 10816 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:49.838824 10816 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:49.839512 10816 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:49.839679 10816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:49.840237 10816 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:49.841167 10816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 20:03:49.841213 10816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.841265 10816 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 20:03:49.841290 10816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:49.849834 10816 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:40257
I20250628 20:03:49.849869 10943 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:40257 every 8 connection(s)
I20250628 20:03:49.850841 10816 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/info.pb
I20250628 20:03:49.854888 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 10816
I20250628 20:03:49.854990 5521 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal/instance
I20250628 20:03:49.859664 10944 heartbeater.cc:344] Connected to a master server at 127.5.100.126:35675
I20250628 20:03:49.859954 10944 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:49.860584 10944 heartbeater.cc:507] Master 127.5.100.126:35675 requested a full tablet report, sending...
I20250628 20:03:49.862828 10494 ts_manager.cc:194] Registered new tserver with Master: 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67:40257)
I20250628 20:03:49.863385 10494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:53759
I20250628 20:03:49.868736 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:49.875370 5521 test_util.cc:276] Using random seed: 1258917543
I20250628 20:03:49.884159 10494 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43660:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250628 20:03:49.885215 10494 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 20:03:49.900974 10747 tablet_service.cc:1468] Processing CreateTablet for tablet 84f8459fd6f7440696271de2b044b209 (DEFAULT_TABLE table=pre_rebuild [id=81ea671d652e4ef6962563d0e6715a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:49.900902 10878 tablet_service.cc:1468] Processing CreateTablet for tablet 84f8459fd6f7440696271de2b044b209 (DEFAULT_TABLE table=pre_rebuild [id=81ea671d652e4ef6962563d0e6715a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:49.901245 10616 tablet_service.cc:1468] Processing CreateTablet for tablet 84f8459fd6f7440696271de2b044b209 (DEFAULT_TABLE table=pre_rebuild [id=81ea671d652e4ef6962563d0e6715a8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:49.901962 10747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f8459fd6f7440696271de2b044b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:49.901962 10878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f8459fd6f7440696271de2b044b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:49.902146 10616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f8459fd6f7440696271de2b044b209. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:49.906762 10966 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Bootstrap starting.
I20250628 20:03:49.908174 10968 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Bootstrap starting.
I20250628 20:03:49.909093 10966 tablet_bootstrap.cc:654] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:49.909821 10967 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Bootstrap starting.
I20250628 20:03:49.909960 10966 log.cc:826] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:49.910205 10968 tablet_bootstrap.cc:654] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:49.910863 10968 log.cc:826] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:49.911643 10967 tablet_bootstrap.cc:654] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:49.911773 10966 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: No bootstrap required, opened a new log
I20250628 20:03:49.911998 10966 ts_tablet_manager.cc:1397] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250628 20:03:49.912322 10967 log.cc:826] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:49.912554 10968 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: No bootstrap required, opened a new log
I20250628 20:03:49.912721 10968 ts_tablet_manager.cc:1397] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250628 20:03:49.913617 10967 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: No bootstrap required, opened a new log
I20250628 20:03:49.913815 10967 ts_tablet_manager.cc:1397] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250628 20:03:49.914994 10966 raft_consensus.cc:357] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.915225 10966 raft_consensus.cc:383] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:49.915309 10966 raft_consensus.cc:738] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ddd3063d4842e790d3a80c6f0d4f7c, State: Initialized, Role: FOLLOWER
I20250628 20:03:49.915514 10968 raft_consensus.cc:357] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.915696 10968 raft_consensus.cc:383] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:49.915776 10968 raft_consensus.cc:738] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a585f79fd34be99b0e3b0c0c0faf15, State: Initialized, Role: FOLLOWER
I20250628 20:03:49.915748 10966 consensus_queue.cc:260] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.916195 10968 consensus_queue.cc:260] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.916325 10967 raft_consensus.cc:357] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.916558 10967 raft_consensus.cc:383] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:49.916661 10967 raft_consensus.cc:738] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca3462212d04f1aafd36a29e9b20ec6, State: Initialized, Role: FOLLOWER
I20250628 20:03:49.916858 10944 heartbeater.cc:499] Master 127.5.100.126:35675 was elected leader, sending a full tablet report...
I20250628 20:03:49.916970 10968 ts_tablet_manager.cc:1428] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 20:03:49.917052 10967 consensus_queue.cc:260] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.917677 10813 heartbeater.cc:499] Master 127.5.100.126:35675 was elected leader, sending a full tablet report...
I20250628 20:03:49.917747 10967 ts_tablet_manager.cc:1428] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 20:03:49.916695 10966 ts_tablet_manager.cc:1428] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250628 20:03:49.918181 10682 heartbeater.cc:499] Master 127.5.100.126:35675 was elected leader, sending a full tablet report...
W20250628 20:03:49.920717 10814 tablet.cc:2378] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:49.942442 10972 raft_consensus.cc:491] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:49.942637 10972 raft_consensus.cc:513] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.943744 10972 leader_election.cc:290] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981), 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67:40257)
I20250628 20:03:49.947937 10898 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f8459fd6f7440696271de2b044b209" candidate_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" is_pre_election: true
I20250628 20:03:49.948241 10898 raft_consensus.cc:2466] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81ddd3063d4842e790d3a80c6f0d4f7c in term 0.
I20250628 20:03:49.948351 10767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f8459fd6f7440696271de2b044b209" candidate_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca3462212d04f1aafd36a29e9b20ec6" is_pre_election: true
I20250628 20:03:49.948606 10767 raft_consensus.cc:2466] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81ddd3063d4842e790d3a80c6f0d4f7c in term 0.
I20250628 20:03:49.948679 10570 leader_election.cc:304] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81ddd3063d4842e790d3a80c6f0d4f7c, 88a585f79fd34be99b0e3b0c0c0faf15; no voters:
I20250628 20:03:49.948953 10972 raft_consensus.cc:2802] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 20:03:49.949043 10972 raft_consensus.cc:491] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 20:03:49.949101 10972 raft_consensus.cc:3058] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:49.950050 10972 raft_consensus.cc:513] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.950395 10972 leader_election.cc:290] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [CANDIDATE]: Term 1 election: Requested vote from peers fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981), 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67:40257)
I20250628 20:03:49.950714 10767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f8459fd6f7440696271de2b044b209" candidate_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca3462212d04f1aafd36a29e9b20ec6"
I20250628 20:03:49.950740 10898 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f8459fd6f7440696271de2b044b209" candidate_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a585f79fd34be99b0e3b0c0c0faf15"
I20250628 20:03:49.950824 10767 raft_consensus.cc:3058] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:49.950836 10898 raft_consensus.cc:3058] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:49.951653 10767 raft_consensus.cc:2466] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81ddd3063d4842e790d3a80c6f0d4f7c in term 1.
I20250628 20:03:49.951973 10569 leader_election.cc:304] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81ddd3063d4842e790d3a80c6f0d4f7c, fca3462212d04f1aafd36a29e9b20ec6; no voters:
I20250628 20:03:49.952078 10898 raft_consensus.cc:2466] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81ddd3063d4842e790d3a80c6f0d4f7c in term 1.
I20250628 20:03:49.952155 10972 raft_consensus.cc:2802] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:49.952381 10972 raft_consensus.cc:695] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 LEADER]: Becoming Leader. State: Replica: 81ddd3063d4842e790d3a80c6f0d4f7c, State: Running, Role: LEADER
I20250628 20:03:49.952718 10972 consensus_queue.cc:237] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:49.955734 10493 catalog_manager.cc:5582] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 81ddd3063d4842e790d3a80c6f0d4f7c (127.5.100.65). New cstate: current_term: 1 leader_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:49.978533 10898 raft_consensus.cc:1273] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 FOLLOWER]: Refusing update from remote peer 81ddd3063d4842e790d3a80c6f0d4f7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 20:03:49.978533 10767 raft_consensus.cc:1273] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 1 FOLLOWER]: Refusing update from remote peer 81ddd3063d4842e790d3a80c6f0d4f7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 20:03:49.979265 10976 consensus_queue.cc:1035] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:49.979427 10972 consensus_queue.cc:1035] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:49.986965 10984 mvcc.cc:204] Tried to move back new op lower bound from 7172673658792198144 to 7172673658688974848. Current Snapshot: MvccSnapshot[applied={T|T < 7172673658785284096}]
I20250628 20:03:49.991590 10985 mvcc.cc:204] Tried to move back new op lower bound from 7172673658792198144 to 7172673658688974848. Current Snapshot: MvccSnapshot[applied={T|T < 7172673658785284096}]
I20250628 20:03:50.823796 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 10463
W20250628 20:03:50.939314 11006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:50.939563 11006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:50.942708 11006 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250628 20:03:50.975903 11006 minidump.cc:252] Setting minidump size limit to 20M
I20250628 20:03:50.976850 11006 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:50.977587 11006 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:50.981930 11029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:50.981941 11030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:50.981937 11032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:50.982008 11006 server_base.cc:1048] running on GCE node
I20250628 20:03:50.982545 11006 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250628 20:03:50.982712 11006 hybrid_clock.cc:648] HybridClock initialized: now 1751141030982706 us; error 82340 us; skew 500 ppm
I20250628 20:03:50.984171 11006 webserver.cc:469] Webserver started at http://0.0.0.0:42253/ using document root <none> and password file <none>
I20250628 20:03:50.984699 11006 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:50.984763 11006 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:50.984910 11006 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 20:03:50.986425 11006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/instance:
uuid: "8a7e95cf07134e308fb8ff223c8865bb"
format_stamp: "Formatted at 2025-06-28 20:03:50 on dist-test-slave-rmcj"
I20250628 20:03:50.986850 11006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal/instance:
uuid: "8a7e95cf07134e308fb8ff223c8865bb"
format_stamp: "Formatted at 2025-06-28 20:03:50 on dist-test-slave-rmcj"
W20250628 20:03:50.989037 10682 heartbeater.cc:646] Failed to heartbeat to 127.5.100.126:35675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.100.126:35675: connect: Connection refused (error 111)
W20250628 20:03:50.989899 10813 heartbeater.cc:646] Failed to heartbeat to 127.5.100.126:35675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.100.126:35675: connect: Connection refused (error 111)
W20250628 20:03:50.990438 10944 heartbeater.cc:646] Failed to heartbeat to 127.5.100.126:35675 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.100.126:35675: connect: Connection refused (error 111)
I20250628 20:03:50.990518 11006 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250628 20:03:50.992563 11040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:50.993248 11006 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:50.993351 11006 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
uuid: "8a7e95cf07134e308fb8ff223c8865bb"
format_stamp: "Formatted at 2025-06-28 20:03:50 on dist-test-slave-rmcj"
I20250628 20:03:50.993444 11006 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:51.051738 11006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:51.052377 11006 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:51.052578 11006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:51.054586 11006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:51.059484 11006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Bootstrap starting.
I20250628 20:03:51.061479 11006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:51.062492 11006 log.cc:826] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:51.064256 11006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: No bootstrap required, opened a new log
I20250628 20:03:51.066743 11006 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER }
I20250628 20:03:51.066926 11006 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:51.066977 11006 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7e95cf07134e308fb8ff223c8865bb, State: Initialized, Role: FOLLOWER
I20250628 20:03:51.067431 11006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER }
I20250628 20:03:51.067548 11006 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:51.067605 11006 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:51.067708 11006 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:51.068476 11006 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER }
I20250628 20:03:51.068806 11006 leader_election.cc:304] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7e95cf07134e308fb8ff223c8865bb; no voters:
I20250628 20:03:51.069090 11006 leader_election.cc:290] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 20:03:51.069228 11047 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:51.069459 11047 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 LEADER]: Becoming Leader. State: Replica: 8a7e95cf07134e308fb8ff223c8865bb, State: Running, Role: LEADER
I20250628 20:03:51.069840 11047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER }
I20250628 20:03:51.071657 11049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7e95cf07134e308fb8ff223c8865bb. Latest consensus state: current_term: 1 leader_uuid: "8a7e95cf07134e308fb8ff223c8865bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER } }
I20250628 20:03:51.071689 11048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7e95cf07134e308fb8ff223c8865bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER } }
I20250628 20:03:51.071853 11049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:51.071853 11048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:51.076661 11006 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: stopping tablet replica
I20250628 20:03:51.076903 11006 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 LEADER]: Raft consensus shutting down.
I20250628 20:03:51.077034 11006 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 20:03:51.078078 11006 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250628 20:03:51.078322 11006 acceptor_pool.cc:194] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250628 20:03:51.089532 11006 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250628 20:03:51.099665 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 10554
I20250628 20:03:51.105705 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 10685
I20250628 20:03:51.110847 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 10816
I20250628 20:03:51.116158 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:35675
--webserver_interface=127.5.100.126
--webserver_port=46205
--builtin_ntp_servers=127.5.100.84:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.100.126:35675 with env {}
W20250628 20:03:51.220386 11053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:51.220620 11053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:51.220659 11053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:51.223969 11053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 20:03:51.224050 11053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:51.224071 11053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 20:03:51.224088 11053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 20:03:51.228108 11053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.100.126:35675
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.100.126:35675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.100.126
--webserver_port=46205
--never_fsync=true
--heap_profile_path=/tmp/kudu.11053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:51.229379 11053 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:51.230623 11053 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:51.237357 11059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:51.237357 11058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:51.237357 11061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:51.237569 11053 server_base.cc:1048] running on GCE node
I20250628 20:03:51.238189 11053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:51.239156 11053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:51.240316 11053 hybrid_clock.cc:648] HybridClock initialized: now 1751141031240296 us; error 34 us; skew 500 ppm
I20250628 20:03:51.242257 11053 webserver.cc:469] Webserver started at http://127.5.100.126:46205/ using document root <none> and password file <none>
I20250628 20:03:51.242820 11053 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:51.242893 11053 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:51.246343 11053 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:51.248274 11067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:51.249354 11053 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:51.249462 11053 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
uuid: "8a7e95cf07134e308fb8ff223c8865bb"
format_stamp: "Formatted at 2025-06-28 20:03:50 on dist-test-slave-rmcj"
I20250628 20:03:51.250069 11053 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:51.271242 11053 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:51.271941 11053 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:51.272125 11053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:51.279244 11053 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.126:35675
I20250628 20:03:51.279250 11119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.126:35675 every 8 connection(s)
I20250628 20:03:51.280426 11053 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/info.pb
I20250628 20:03:51.281862 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 11053
I20250628 20:03:51.282286 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.65:38815
--local_ip_for_outbound_sockets=127.5.100.65
--tserver_master_addrs=127.5.100.126:35675
--webserver_port=46597
--webserver_interface=127.5.100.65
--builtin_ntp_servers=127.5.100.84:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 20:03:51.284778 11120 sys_catalog.cc:263] Verifying existing consensus state
I20250628 20:03:51.288430 11120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Bootstrap starting.
I20250628 20:03:51.292785 11120 log.cc:826] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:51.298245 11120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:51.298578 11120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Bootstrap complete.
I20250628 20:03:51.301257 11120 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } }
I20250628 20:03:51.301455 11120 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7e95cf07134e308fb8ff223c8865bb, State: Initialized, Role: FOLLOWER
I20250628 20:03:51.301937 11120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } }
I20250628 20:03:51.302057 11120 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 20:03:51.302116 11120 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 20:03:51.302208 11120 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:51.302965 11120 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } }
I20250628 20:03:51.303257 11120 leader_election.cc:304] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7e95cf07134e308fb8ff223c8865bb; no voters:
I20250628 20:03:51.303499 11120 leader_election.cc:290] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 20:03:51.303602 11124 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 2 FOLLOWER]: Leader election won for term 2
I20250628 20:03:51.303766 11124 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [term 2 LEADER]: Becoming Leader. State: Replica: 8a7e95cf07134e308fb8ff223c8865bb, State: Running, Role: LEADER
I20250628 20:03:51.304090 11124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } }
I20250628 20:03:51.304615 11120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 20:03:51.304970 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7e95cf07134e308fb8ff223c8865bb. Latest consensus state: current_term: 2 leader_uuid: "8a7e95cf07134e308fb8ff223c8865bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } } }
I20250628 20:03:51.305157 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:51.305322 11125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8a7e95cf07134e308fb8ff223c8865bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7e95cf07134e308fb8ff223c8865bb" member_type: VOTER last_known_addr { host: "127.5.100.126" port: 35675 } } }
I20250628 20:03:51.305487 11125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb [sys.catalog]: This master's current role is: LEADER
I20250628 20:03:51.305549 11128 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 20:03:51.309232 11128 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=420b888392b64acf8f332b9cf62337f8]
I20250628 20:03:51.310477 11128 tablet_loader.cc:96] loaded metadata for tablet 84f8459fd6f7440696271de2b044b209 (table pre_rebuild [id=420b888392b64acf8f332b9cf62337f8])
I20250628 20:03:51.310619 11128 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 20:03:51.314509 11128 catalog_manager.cc:1349] Generated new cluster ID: 5b1ef7084f4d4582abec1c35dbf5ee2c
I20250628 20:03:51.314571 11128 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 20:03:51.335471 11128 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 20:03:51.336661 11128 catalog_manager.cc:1506] Loading token signing keys...
I20250628 20:03:51.347772 11128 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Generated new TSK 0
I20250628 20:03:51.348531 11128 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 20:03:51.391993 11122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:51.392221 11122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:51.392282 11122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:51.395470 11122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:51.395599 11122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.65
I20250628 20:03:51.399331 11122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.65:38815
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.100.65
--webserver_port=46597
--tserver_master_addrs=127.5.100.126:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.11122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.65
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:51.400429 11122 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:51.401575 11122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:51.409479 11150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:51.409528 11147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:51.409509 11148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:51.410177 11122 server_base.cc:1048] running on GCE node
I20250628 20:03:51.410526 11122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:51.411108 11122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:51.412259 11122 hybrid_clock.cc:648] HybridClock initialized: now 1751141031412236 us; error 40 us; skew 500 ppm
I20250628 20:03:51.414198 11122 webserver.cc:469] Webserver started at http://127.5.100.65:46597/ using document root <none> and password file <none>
I20250628 20:03:51.414749 11122 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:51.414822 11122 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:51.418138 11122 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:51.420101 11156 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:51.421129 11122 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:51.421227 11122 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
uuid: "81ddd3063d4842e790d3a80c6f0d4f7c"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:51.421607 11122 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:51.436911 11122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:51.437561 11122 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:51.437758 11122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:51.438371 11122 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:51.439828 11163 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250628 20:03:51.443059 11122 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 20:03:51.443131 11122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20250628 20:03:51.443185 11122 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 20:03:51.444684 11122 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 20:03:51.444738 11122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250628 20:03:51.445070 11163 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Bootstrap starting.
I20250628 20:03:51.456362 11122 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.65:38815
I20250628 20:03:51.456398 11270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.65:38815 every 8 connection(s)
I20250628 20:03:51.457677 11122 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/info.pb
I20250628 20:03:51.458475 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 11122
I20250628 20:03:51.458923 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.66:44981
--local_ip_for_outbound_sockets=127.5.100.66
--tserver_master_addrs=127.5.100.126:35675
--webserver_port=34971
--webserver_interface=127.5.100.66
--builtin_ntp_servers=127.5.100.84:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 20:03:51.476450 11271 heartbeater.cc:344] Connected to a master server at 127.5.100.126:35675
I20250628 20:03:51.476758 11271 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:51.477427 11271 heartbeater.cc:507] Master 127.5.100.126:35675 requested a full tablet report, sending...
I20250628 20:03:51.480146 11084 ts_manager.cc:194] Registered new tserver with Master: 81ddd3063d4842e790d3a80c6f0d4f7c (127.5.100.65:38815)
I20250628 20:03:51.482847 11084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.65:34949
I20250628 20:03:51.488636 11163 log.cc:826] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Log is configured to *not* fsync() on all Append() calls
W20250628 20:03:51.574116 11273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:51.574349 11273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:51.574411 11273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:51.577652 11273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:51.577802 11273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.66
I20250628 20:03:51.581513 11273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.66:44981
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.100.66
--webserver_port=34971
--tserver_master_addrs=127.5.100.126:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.11273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.66
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:51.582614 11273 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:51.583693 11273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:51.591990 11282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:51.592001 11281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:51.591990 11284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:51.592211 11273 server_base.cc:1048] running on GCE node
I20250628 20:03:51.592690 11273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:51.593253 11273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:51.594395 11273 hybrid_clock.cc:648] HybridClock initialized: now 1751141031594373 us; error 39 us; skew 500 ppm
I20250628 20:03:51.596216 11273 webserver.cc:469] Webserver started at http://127.5.100.66:34971/ using document root <none> and password file <none>
I20250628 20:03:51.596779 11273 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:51.596860 11273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:51.600407 11273 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:51.602351 11290 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:51.603386 11273 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250628 20:03:51.603502 11273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
uuid: "fca3462212d04f1aafd36a29e9b20ec6"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:51.603878 11273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:51.617619 11273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:51.618283 11273 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:51.618439 11273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:51.618966 11273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:51.620137 11297 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250628 20:03:51.624051 11273 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 20:03:51.624114 11273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250628 20:03:51.624184 11273 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 20:03:51.625612 11273 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 20:03:51.625660 11273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250628 20:03:51.626026 11297 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Bootstrap starting.
I20250628 20:03:51.635771 11273 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.66:44981
I20250628 20:03:51.636236 11404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.66:44981 every 8 connection(s)
I20250628 20:03:51.636904 11273 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/info.pb
I20250628 20:03:51.638157 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 11273
I20250628 20:03:51.638607 5521 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskH2vkox/build/debug/bin/kudu
/tmp/dist-test-taskH2vkox/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.100.67:40257
--local_ip_for_outbound_sockets=127.5.100.67
--tserver_master_addrs=127.5.100.126:35675
--webserver_port=44775
--webserver_interface=127.5.100.67
--builtin_ntp_servers=127.5.100.84:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 20:03:51.647372 11405 heartbeater.cc:344] Connected to a master server at 127.5.100.126:35675
I20250628 20:03:51.647665 11405 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:51.648360 11405 heartbeater.cc:507] Master 127.5.100.126:35675 requested a full tablet report, sending...
I20250628 20:03:51.650425 11084 ts_manager.cc:194] Registered new tserver with Master: fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981)
I20250628 20:03:51.651402 11084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.66:43373
I20250628 20:03:51.666641 11297 log.cc:826] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Log is configured to *not* fsync() on all Append() calls
W20250628 20:03:51.747483 11408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 20:03:51.747700 11408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 20:03:51.747761 11408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 20:03:51.750990 11408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 20:03:51.751133 11408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.100.67
I20250628 20:03:51.754925 11408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.100.84:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.100.67:40257
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.100.67
--webserver_port=44775
--tserver_master_addrs=127.5.100.126:35675
--never_fsync=true
--heap_profile_path=/tmp/kudu.11408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.100.67
--log_dir=/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type DEBUG
built by None at 28 Jun 2025 19:43:16 UTC on 24a791456cd2
build id 6814
I20250628 20:03:51.756067 11408 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 20:03:51.757237 11408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 20:03:51.764628 11415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:51.764835 11408 server_base.cc:1048] running on GCE node
W20250628 20:03:51.764777 11417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 20:03:51.764587 11414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 20:03:51.765555 11408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 20:03:51.766189 11408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 20:03:51.767328 11408 hybrid_clock.cc:648] HybridClock initialized: now 1751141031767305 us; error 33 us; skew 500 ppm
I20250628 20:03:51.769133 11408 webserver.cc:469] Webserver started at http://127.5.100.67:44775/ using document root <none> and password file <none>
I20250628 20:03:51.769727 11408 fs_manager.cc:362] Metadata directory not provided
I20250628 20:03:51.769810 11408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 20:03:51.774834 11408 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20250628 20:03:51.776780 11423 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 20:03:51.777920 11408 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250628 20:03:51.778024 11408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data,/tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
uuid: "88a585f79fd34be99b0e3b0c0c0faf15"
format_stamp: "Formatted at 2025-06-28 20:03:49 on dist-test-slave-rmcj"
I20250628 20:03:51.778417 11408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 20:03:51.791728 11408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 20:03:51.792371 11408 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 20:03:51.792537 11408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 20:03:51.793100 11408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 20:03:51.794348 11430 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250628 20:03:51.797412 11408 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 20:03:51.797472 11408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.001s
I20250628 20:03:51.797544 11408 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 20:03:51.799081 11408 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 20:03:51.799139 11408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250628 20:03:51.799558 11430 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Bootstrap starting.
I20250628 20:03:51.809638 11408 rpc_server.cc:307] RPC server started. Bound to: 127.5.100.67:40257
I20250628 20:03:51.810887 11408 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/info.pb
I20250628 20:03:51.815361 11537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.100.67:40257 every 8 connection(s)
I20250628 20:03:51.818850 5521 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskH2vkox/build/debug/bin/kudu as pid 11408
I20250628 20:03:51.838551 11538 heartbeater.cc:344] Connected to a master server at 127.5.100.126:35675
I20250628 20:03:51.838878 11538 heartbeater.cc:461] Registering TS with master...
I20250628 20:03:51.839649 11538 heartbeater.cc:507] Master 127.5.100.126:35675 requested a full tablet report, sending...
I20250628 20:03:51.840767 11430 log.cc:826] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Log is configured to *not* fsync() on all Append() calls
I20250628 20:03:51.842099 11084 ts_manager.cc:194] Registered new tserver with Master: 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67:40257)
I20250628 20:03:51.843077 11084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.100.67:58489
I20250628 20:03:51.844785 5521 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 20:03:51.863324 11163 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=207 overwritten=0 applied=207 ignored=0} inserts{seen=10300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:51.863768 11163 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Bootstrap complete.
I20250628 20:03:51.865072 11163 ts_tablet_manager.cc:1397] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Time spent bootstrapping tablet: real 0.420s user 0.399s sys 0.012s
I20250628 20:03:51.867846 11163 raft_consensus.cc:357] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:51.868342 11163 raft_consensus.cc:738] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ddd3063d4842e790d3a80c6f0d4f7c, State: Initialized, Role: FOLLOWER
I20250628 20:03:51.868939 11163 consensus_queue.cc:260] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 207, Last appended: 1.207, Last appended by leader: 207, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:51.870014 11163 ts_tablet_manager.cc:1428] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20250628 20:03:51.870350 11271 heartbeater.cc:499] Master 127.5.100.126:35675 was elected leader, sending a full tablet report...
I20250628 20:03:52.021136 11297 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Bootstrap replayed 1/1 log segments. Stats: ops{read=207 overwritten=0 applied=207 ignored=0} inserts{seen=10300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:52.021525 11297 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Bootstrap complete.
I20250628 20:03:52.022461 11297 ts_tablet_manager.cc:1397] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Time spent bootstrapping tablet: real 0.397s user 0.366s sys 0.028s
I20250628 20:03:52.024566 11297 raft_consensus.cc:357] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:52.024893 11297 raft_consensus.cc:738] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca3462212d04f1aafd36a29e9b20ec6, State: Initialized, Role: FOLLOWER
I20250628 20:03:52.025341 11297 consensus_queue.cc:260] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 207, Last appended: 1.207, Last appended by leader: 207, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:52.026083 11405 heartbeater.cc:499] Master 127.5.100.126:35675 was elected leader, sending a full tablet report...
I20250628 20:03:52.026245 11297 ts_tablet_manager.cc:1428] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 20:03:52.133195 11430 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Bootstrap replayed 1/1 log segments. Stats: ops{read=207 overwritten=0 applied=207 ignored=0} inserts{seen=10300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 20:03:52.133567 11430 tablet_bootstrap.cc:492] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Bootstrap complete.
I20250628 20:03:52.134472 11430 ts_tablet_manager.cc:1397] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Time spent bootstrapping tablet: real 0.335s user 0.323s sys 0.008s
I20250628 20:03:52.136515 11430 raft_consensus.cc:357] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:52.136860 11430 raft_consensus.cc:738] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a585f79fd34be99b0e3b0c0c0faf15, State: Initialized, Role: FOLLOWER
I20250628 20:03:52.137293 11430 consensus_queue.cc:260] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 207, Last appended: 1.207, Last appended by leader: 207, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:52.138024 11538 heartbeater.cc:499] Master 127.5.100.126:35675 was elected leader, sending a full tablet report...
I20250628 20:03:52.138178 11430 ts_tablet_manager.cc:1428] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 20:03:53.459606 11552 raft_consensus.cc:491] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:53.459810 11552 raft_consensus.cc:513] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:53.460911 11552 leader_election.cc:290] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981), 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67:40257)
I20250628 20:03:53.466775 11359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f8459fd6f7440696271de2b044b209" candidate_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 207 } } ignore_live_leader: false dest_uuid: "fca3462212d04f1aafd36a29e9b20ec6" is_pre_election: true
I20250628 20:03:53.466785 11492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f8459fd6f7440696271de2b044b209" candidate_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 207 } } ignore_live_leader: false dest_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" is_pre_election: true
I20250628 20:03:53.467072 11492 raft_consensus.cc:2466] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81ddd3063d4842e790d3a80c6f0d4f7c in term 1.
I20250628 20:03:53.467077 11359 raft_consensus.cc:2466] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81ddd3063d4842e790d3a80c6f0d4f7c in term 1.
I20250628 20:03:53.468353 11157 leader_election.cc:304] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81ddd3063d4842e790d3a80c6f0d4f7c, fca3462212d04f1aafd36a29e9b20ec6; no voters:
I20250628 20:03:53.468602 11552 raft_consensus.cc:2802] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250628 20:03:53.468672 11552 raft_consensus.cc:491] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 20:03:53.468699 11552 raft_consensus.cc:3058] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:53.470422 11552 raft_consensus.cc:513] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:53.470778 11552 leader_election.cc:290] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [CANDIDATE]: Term 2 election: Requested vote from peers fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981), 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67:40257)
I20250628 20:03:53.471150 11359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f8459fd6f7440696271de2b044b209" candidate_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 207 } } ignore_live_leader: false dest_uuid: "fca3462212d04f1aafd36a29e9b20ec6"
I20250628 20:03:53.471168 11492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84f8459fd6f7440696271de2b044b209" candidate_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 207 } } ignore_live_leader: false dest_uuid: "88a585f79fd34be99b0e3b0c0c0faf15"
I20250628 20:03:53.471271 11492 raft_consensus.cc:3058] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:53.471282 11359 raft_consensus.cc:3058] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 1 FOLLOWER]: Advancing to term 2
I20250628 20:03:53.472839 11492 raft_consensus.cc:2466] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81ddd3063d4842e790d3a80c6f0d4f7c in term 2.
I20250628 20:03:53.473156 11158 leader_election.cc:304] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81ddd3063d4842e790d3a80c6f0d4f7c, 88a585f79fd34be99b0e3b0c0c0faf15; no voters:
I20250628 20:03:53.473232 11359 raft_consensus.cc:2466] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81ddd3063d4842e790d3a80c6f0d4f7c in term 2.
I20250628 20:03:53.473320 11552 raft_consensus.cc:2802] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 2 FOLLOWER]: Leader election won for term 2
I20250628 20:03:53.473462 11552 raft_consensus.cc:695] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 2 LEADER]: Becoming Leader. State: Replica: 81ddd3063d4842e790d3a80c6f0d4f7c, State: Running, Role: LEADER
I20250628 20:03:53.473766 11552 consensus_queue.cc:237] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 1.207, Last appended by leader: 207, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:53.476449 11084 catalog_manager.cc:5582] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c reported cstate change: term changed from 0 to 2, leader changed from <none> to 81ddd3063d4842e790d3a80c6f0d4f7c (127.5.100.65), VOTER 81ddd3063d4842e790d3a80c6f0d4f7c (127.5.100.65) added, VOTER 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67) added, VOTER fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66) added. New cstate: current_term: 2 leader_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:53.895488 11359 raft_consensus.cc:1273] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 2 FOLLOWER]: Refusing update from remote peer 81ddd3063d4842e790d3a80c6f0d4f7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20250628 20:03:53.896134 11552 consensus_queue.cc:1035] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20250628 20:03:53.900245 11492 raft_consensus.cc:1273] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 2 FOLLOWER]: Refusing update from remote peer 81ddd3063d4842e790d3a80c6f0d4f7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 2 index: 208. (index mismatch)
I20250628 20:03:53.900816 11552 consensus_queue.cc:1035] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s
I20250628 20:03:53.908775 11225 consensus_queue.cc:237] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 208, Committed index: 208, Last appended: 2.208, Last appended by leader: 207, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:53.909844 11359 raft_consensus.cc:1273] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 2 FOLLOWER]: Refusing update from remote peer 81ddd3063d4842e790d3a80c6f0d4f7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 208. Preceding OpId from leader: term: 2 index: 209. (index mismatch)
I20250628 20:03:53.910197 11552 consensus_queue.cc:1035] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 209, Last known committed idx: 208, Time since last communication: 0.000s
I20250628 20:03:53.911662 11557 raft_consensus.cc:2953] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 2 LEADER]: Committing config change with OpId 2.209: config changed from index -1 to 209, VOTER 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } }
I20250628 20:03:53.912005 11359 raft_consensus.cc:2953] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 2 FOLLOWER]: Committing config change with OpId 2.209: config changed from index -1 to 209, VOTER 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67) evicted. New config: { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } }
I20250628 20:03:53.913640 11071 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 84f8459fd6f7440696271de2b044b209 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 20:03:53.914422 11083 catalog_manager.cc:5582] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 reported cstate change: config changed from index -1 to 209, VOTER 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67) evicted. New cstate: current_term: 2 leader_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" committed_config { opid_index: 209 OBSOLETE_local: false peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } }
I20250628 20:03:53.917814 11225 consensus_queue.cc:237] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 2.209, Last appended by leader: 207, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } }
I20250628 20:03:53.918501 11557 raft_consensus.cc:2953] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 2 LEADER]: Committing config change with OpId 2.210: config changed from index 209 to 210, VOTER fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66) evicted. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } }
I20250628 20:03:53.919672 11071 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 84f8459fd6f7440696271de2b044b209 with cas_config_opid_index 209: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 20:03:53.920177 11472 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f8459fd6f7440696271de2b044b209 with delete_type TABLET_DATA_TOMBSTONED (TS 88a585f79fd34be99b0e3b0c0c0faf15 not found in new config with opid_index 209) from {username='slave'} at 127.0.0.1:55054
I20250628 20:03:53.920523 11084 catalog_manager.cc:5582] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c reported cstate change: config changed from index 209 to 210, VOTER fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66) evicted. New cstate: current_term: 2 leader_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } health_report { overall_health: HEALTHY } } }
I20250628 20:03:53.920727 11572 tablet_replica.cc:331] stopping tablet replica
I20250628 20:03:53.921056 11572 raft_consensus.cc:2241] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250628 20:03:53.921296 11572 raft_consensus.cc:2270] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 20:03:53.922777 11572 ts_tablet_manager.cc:1905] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 20:03:53.925843 11572 ts_tablet_manager.cc:1918] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.208
I20250628 20:03:53.925931 11572 log.cc:1199] T 84f8459fd6f7440696271de2b044b209 P 88a585f79fd34be99b0e3b0c0c0faf15: Deleting WAL directory at /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/wal/wals/84f8459fd6f7440696271de2b044b209
I20250628 20:03:53.926185 11337 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f8459fd6f7440696271de2b044b209 with delete_type TABLET_DATA_TOMBSTONED (TS fca3462212d04f1aafd36a29e9b20ec6 not found in new config with opid_index 210) from {username='slave'} at 127.0.0.1:34284
I20250628 20:03:53.926604 11069 catalog_manager.cc:4928] TS 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67:40257): tablet 84f8459fd6f7440696271de2b044b209 (table pre_rebuild [id=420b888392b64acf8f332b9cf62337f8]) successfully deleted
I20250628 20:03:53.926710 11573 tablet_replica.cc:331] stopping tablet replica
I20250628 20:03:53.926956 11573 raft_consensus.cc:2241] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250628 20:03:53.927193 11573 raft_consensus.cc:2270] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 20:03:53.928398 11573 ts_tablet_manager.cc:1905] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 20:03:53.930529 11573 ts_tablet_manager.cc:1918] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.209
I20250628 20:03:53.930608 11573 log.cc:1199] T 84f8459fd6f7440696271de2b044b209 P fca3462212d04f1aafd36a29e9b20ec6: Deleting WAL directory at /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/wal/wals/84f8459fd6f7440696271de2b044b209
I20250628 20:03:53.931252 11068 catalog_manager.cc:4928] TS fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981): tablet 84f8459fd6f7440696271de2b044b209 (table pre_rebuild [id=420b888392b64acf8f332b9cf62337f8]) successfully deleted
W20250628 20:03:54.004935 5521 scanner-internal.cc:458] Time spent opening tablet: real 2.145s user 0.003s sys 0.000s
I20250628 20:03:54.122339 11472 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:54.124563 11200 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:54.130026 11337 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8a7e95cf07134e308fb8ff223c8865bb | 127.5.100.126:35675 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.100.84:34675 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
81ddd3063d4842e790d3a80c6f0d4f7c | 127.5.100.65:38815 | HEALTHY | <none> | 1 | 0
88a585f79fd34be99b0e3b0c0c0faf15 | 127.5.100.67:40257 | HEALTHY | <none> | 0 | 0
fca3462212d04f1aafd36a29e9b20ec6 | 127.5.100.66:44981 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.100.65 | experimental | 127.5.100.65:38815
local_ip_for_outbound_sockets | 127.5.100.66 | experimental | 127.5.100.66:44981
local_ip_for_outbound_sockets | 127.5.100.67 | experimental | 127.5.100.67:40257
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/info.pb | hidden | 127.5.100.65:38815
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/info.pb | hidden | 127.5.100.66:44981
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/info.pb | hidden | 127.5.100.67:40257
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.100.84:34675 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 20:03:54.154332 5521 log_verifier.cc:126] Checking tablet 84f8459fd6f7440696271de2b044b209
I20250628 20:03:54.199259 5521 log_verifier.cc:177] Verified matching terms for 210 ops in tablet 84f8459fd6f7440696271de2b044b209
I20250628 20:03:54.200388 11084 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43672:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250628 20:03:54.200580 11084 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43672:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250628 20:03:54.203493 11084 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 8a7e95cf07134e308fb8ff223c8865bb: Sending DeleteTablet for 1 replicas of tablet 84f8459fd6f7440696271de2b044b209
I20250628 20:03:54.204243 11200 tablet_service.cc:1515] Processing DeleteTablet for tablet 84f8459fd6f7440696271de2b044b209 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-28 20:03:54 UTC) from {username='slave'} at 127.0.0.1:39830
I20250628 20:03:54.204708 11601 tablet_replica.cc:331] stopping tablet replica
I20250628 20:03:54.204759 5521 test_util.cc:276] Using random seed: 1263246932
I20250628 20:03:54.205011 11601 raft_consensus.cc:2241] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 2 LEADER]: Raft consensus shutting down.
I20250628 20:03:54.205302 11601 raft_consensus.cc:2270] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 20:03:54.207607 11601 ts_tablet_manager.cc:1905] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 20:03:54.210569 11601 ts_tablet_manager.cc:1918] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.210
I20250628 20:03:54.210664 11601 log.cc:1199] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Deleting WAL directory at /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/wal/wals/84f8459fd6f7440696271de2b044b209
I20250628 20:03:54.211047 11601 ts_tablet_manager.cc:1939] T 84f8459fd6f7440696271de2b044b209 P 81ddd3063d4842e790d3a80c6f0d4f7c: Deleting consensus metadata
I20250628 20:03:54.212224 11071 catalog_manager.cc:4928] TS 81ddd3063d4842e790d3a80c6f0d4f7c (127.5.100.65:38815): tablet 84f8459fd6f7440696271de2b044b209 (table pre_rebuild [id=420b888392b64acf8f332b9cf62337f8]) successfully deleted
I20250628 20:03:54.213382 11084 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43716:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250628 20:03:54.214636 11084 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 20:03:54.220415 11200 tablet_service.cc:1468] Processing CreateTablet for tablet b60ed6823f2b48469fc85dae92073936 (DEFAULT_TABLE table=post_rebuild [id=f7a3bc2922954e7fa626bbb9830ef1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:54.220559 11337 tablet_service.cc:1468] Processing CreateTablet for tablet b60ed6823f2b48469fc85dae92073936 (DEFAULT_TABLE table=post_rebuild [id=f7a3bc2922954e7fa626bbb9830ef1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:54.220698 11472 tablet_service.cc:1468] Processing CreateTablet for tablet b60ed6823f2b48469fc85dae92073936 (DEFAULT_TABLE table=post_rebuild [id=f7a3bc2922954e7fa626bbb9830ef1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 20:03:54.221047 11337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b60ed6823f2b48469fc85dae92073936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:54.221050 11200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b60ed6823f2b48469fc85dae92073936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:54.221131 11472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b60ed6823f2b48469fc85dae92073936. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 20:03:54.223559 11607 tablet_bootstrap.cc:492] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6: Bootstrap starting.
I20250628 20:03:54.223559 11608 tablet_bootstrap.cc:492] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15: Bootstrap starting.
I20250628 20:03:54.224417 11608 tablet_bootstrap.cc:654] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:54.224417 11607 tablet_bootstrap.cc:654] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:54.225028 11609 tablet_bootstrap.cc:492] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c: Bootstrap starting.
I20250628 20:03:54.225490 11607 tablet_bootstrap.cc:492] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6: No bootstrap required, opened a new log
I20250628 20:03:54.225584 11607 ts_tablet_manager.cc:1397] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250628 20:03:54.226066 11609 tablet_bootstrap.cc:654] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c: Neither blocks nor log segments found. Creating new log.
I20250628 20:03:54.226275 11608 tablet_bootstrap.cc:492] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15: No bootstrap required, opened a new log
I20250628 20:03:54.226227 11607 raft_consensus.cc:357] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.226328 11607 raft_consensus.cc:383] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:54.226353 11608 ts_tablet_manager.cc:1397] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250628 20:03:54.226406 11607 raft_consensus.cc:738] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca3462212d04f1aafd36a29e9b20ec6, State: Initialized, Role: FOLLOWER
I20250628 20:03:54.226560 11607 consensus_queue.cc:260] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.226804 11607 ts_tablet_manager.cc:1428] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 20:03:54.226836 11608 raft_consensus.cc:357] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.226943 11608 raft_consensus.cc:383] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:54.227020 11608 raft_consensus.cc:738] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a585f79fd34be99b0e3b0c0c0faf15, State: Initialized, Role: FOLLOWER
I20250628 20:03:54.227201 11608 consensus_queue.cc:260] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.227527 11608 ts_tablet_manager.cc:1428] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 20:03:54.227548 11609 tablet_bootstrap.cc:492] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c: No bootstrap required, opened a new log
I20250628 20:03:54.227622 11609 ts_tablet_manager.cc:1397] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250628 20:03:54.228132 11609 raft_consensus.cc:357] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.228300 11609 raft_consensus.cc:383] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 20:03:54.228385 11609 raft_consensus.cc:738] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ddd3063d4842e790d3a80c6f0d4f7c, State: Initialized, Role: FOLLOWER
I20250628 20:03:54.228549 11609 consensus_queue.cc:260] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.228793 11609 ts_tablet_manager.cc:1428] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250628 20:03:54.302599 11567 raft_consensus.cc:491] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 20:03:54.302745 11567 raft_consensus.cc:513] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.303803 11567 leader_election.cc:290] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81ddd3063d4842e790d3a80c6f0d4f7c (127.5.100.65:38815), fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981)
I20250628 20:03:54.307475 11359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b60ed6823f2b48469fc85dae92073936" candidate_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca3462212d04f1aafd36a29e9b20ec6" is_pre_election: true
I20250628 20:03:54.307544 11225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b60ed6823f2b48469fc85dae92073936" candidate_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" is_pre_election: true
I20250628 20:03:54.307638 11359 raft_consensus.cc:2466] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a585f79fd34be99b0e3b0c0c0faf15 in term 0.
I20250628 20:03:54.307785 11225 raft_consensus.cc:2466] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a585f79fd34be99b0e3b0c0c0faf15 in term 0.
I20250628 20:03:54.307974 11424 leader_election.cc:304] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a585f79fd34be99b0e3b0c0c0faf15, fca3462212d04f1aafd36a29e9b20ec6; no voters:
I20250628 20:03:54.308207 11567 raft_consensus.cc:2802] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 20:03:54.308269 11567 raft_consensus.cc:491] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 20:03:54.308305 11567 raft_consensus.cc:3058] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:54.309088 11567 raft_consensus.cc:513] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.309435 11567 leader_election.cc:290] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [CANDIDATE]: Term 1 election: Requested vote from peers 81ddd3063d4842e790d3a80c6f0d4f7c (127.5.100.65:38815), fca3462212d04f1aafd36a29e9b20ec6 (127.5.100.66:44981)
I20250628 20:03:54.309820 11359 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b60ed6823f2b48469fc85dae92073936" candidate_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fca3462212d04f1aafd36a29e9b20ec6"
I20250628 20:03:54.309815 11225 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b60ed6823f2b48469fc85dae92073936" candidate_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c"
I20250628 20:03:54.309926 11225 raft_consensus.cc:3058] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:54.309934 11359 raft_consensus.cc:3058] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6 [term 0 FOLLOWER]: Advancing to term 1
I20250628 20:03:54.310654 11225 raft_consensus.cc:2466] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a585f79fd34be99b0e3b0c0c0faf15 in term 1.
I20250628 20:03:54.310660 11359 raft_consensus.cc:2466] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a585f79fd34be99b0e3b0c0c0faf15 in term 1.
I20250628 20:03:54.310921 11427 leader_election.cc:304] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81ddd3063d4842e790d3a80c6f0d4f7c, 88a585f79fd34be99b0e3b0c0c0faf15; no voters:
I20250628 20:03:54.311076 11567 raft_consensus.cc:2802] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 20:03:54.311308 11567 raft_consensus.cc:695] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [term 1 LEADER]: Becoming Leader. State: Replica: 88a585f79fd34be99b0e3b0c0c0faf15, State: Running, Role: LEADER
I20250628 20:03:54.311553 11567 consensus_queue.cc:237] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } }
I20250628 20:03:54.313746 11084 catalog_manager.cc:5582] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a585f79fd34be99b0e3b0c0c0faf15 (127.5.100.67). New cstate: current_term: 1 leader_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a585f79fd34be99b0e3b0c0c0faf15" member_type: VOTER last_known_addr { host: "127.5.100.67" port: 40257 } health_report { overall_health: HEALTHY } } }
W20250628 20:03:54.319170 11539 tablet.cc:2378] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 20:03:54.340909 11359 raft_consensus.cc:1273] T b60ed6823f2b48469fc85dae92073936 P fca3462212d04f1aafd36a29e9b20ec6 [term 1 FOLLOWER]: Refusing update from remote peer 88a585f79fd34be99b0e3b0c0c0faf15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 20:03:54.341074 11225 raft_consensus.cc:1273] T b60ed6823f2b48469fc85dae92073936 P 81ddd3063d4842e790d3a80c6f0d4f7c [term 1 FOLLOWER]: Refusing update from remote peer 88a585f79fd34be99b0e3b0c0c0faf15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 20:03:54.341401 11567 consensus_queue.cc:1035] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fca3462212d04f1aafd36a29e9b20ec6" member_type: VOTER last_known_addr { host: "127.5.100.66" port: 44981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:54.341794 11567 consensus_queue.cc:1035] T b60ed6823f2b48469fc85dae92073936 P 88a585f79fd34be99b0e3b0c0c0faf15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81ddd3063d4842e790d3a80c6f0d4f7c" member_type: VOTER last_known_addr { host: "127.5.100.65" port: 38815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 20:03:54.347908 11566 mvcc.cc:204] Tried to move back new op lower bound from 7172673676661948416 to 7172673676542607360. Current Snapshot: MvccSnapshot[applied={T|T < 7172673676655476736}]
I20250628 20:03:54.349370 11565 mvcc.cc:204] Tried to move back new op lower bound from 7172673676661948416 to 7172673676542607360. Current Snapshot: MvccSnapshot[applied={T|T < 7172673676655476736}]
I20250628 20:03:55.141367 11472 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 20:03:55.141366 11200 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 20:03:55.142853 11337 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8a7e95cf07134e308fb8ff223c8865bb | 127.5.100.126:35675 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.100.84:34675 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
81ddd3063d4842e790d3a80c6f0d4f7c | 127.5.100.65:38815 | HEALTHY | <none> | 0 | 0
88a585f79fd34be99b0e3b0c0c0faf15 | 127.5.100.67:40257 | HEALTHY | <none> | 1 | 0
fca3462212d04f1aafd36a29e9b20ec6 | 127.5.100.66:44981 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.100.65 | experimental | 127.5.100.65:38815
local_ip_for_outbound_sockets | 127.5.100.66 | experimental | 127.5.100.66:44981
local_ip_for_outbound_sockets | 127.5.100.67 | experimental | 127.5.100.67:40257
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-0/data/info.pb | hidden | 127.5.100.65:38815
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-1/data/info.pb | hidden | 127.5.100.66:44981
server_dump_info_path | /tmp/dist-test-taskH2vkox/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751141006956821-5521-0/minicluster-data/ts-2/data/info.pb | hidden | 127.5.100.67:40257
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.100.84:34675 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 20:03:55.163900 5521 log_verifier.cc:126] Checking tablet 84f8459fd6f7440696271de2b044b209
I20250628 20:03:55.164000 5521 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 84f8459fd6f7440696271de2b044b209
I20250628 20:03:55.164029 5521 log_verifier.cc:126] Checking tablet b60ed6823f2b48469fc85dae92073936
I20250628 20:03:55.254916 5521 log_verifier.cc:177] Verified matching terms for 206 ops in tablet b60ed6823f2b48469fc85dae92073936
I20250628 20:03:55.258023 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 11122
I20250628 20:03:55.264797 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 11273
I20250628 20:03:55.270102 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 11408
I20250628 20:03:55.275414 5521 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskH2vkox/build/debug/bin/kudu with pid 11053
2025-06-28T20:03:55Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (6230 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (6230 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (28329 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST